Description of problem:
  Minor update of overcloud in OSP 15 fails during node startup

2019-08-21 18:07:34 |         "Creating systemd unit file: /etc/systemd/system/tripleo_memcached-0vb7evm1.service",
2019-08-21 18:07:34 |         "Executing: systemctl daemon-reload",
2019-08-21 18:07:34 |         "Executing: systemctl enable --now tripleo_memcached-0vb7evm1",
2019-08-21 18:07:34 |         "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_memcached-0vb7evm1.service → /etc/systemd/system/tripleo_memcached-0vb7e
vm1.service.",
2019-08-21 18:07:34 |         "Job for tripleo_memcached-0vb7evm1.service failed because the service did not take the steps required by its unit configuration.",
2019-08-21 18:07:34 |         "See \"systemctl status tripleo_memcached-0vb7evm1.service\" and \"journalctl -xe\" for details.",
2019-08-21 18:07:34 |         "systemctl failed",
2019-08-21 18:07:34 |         "Traceback (most recent call last):",
2019-08-21 18:07:34 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 31, in systemctl",
2019-08-21 18:07:34 |         "    subprocess.check_call(cmd)",
2019-08-21 18:07:34 |         "  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call",
2019-08-21 18:07:34 |         "    raise CalledProcessError(retcode, cmd)",
2019-08-21 18:07:34 |         "subprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_memcached-0vb7evm1']' returned non-zero exit status 1.",
2019-08-21 18:07:34 |         "During handling of the above exception, another exception occurred:",
2019-08-21 18:07:34 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 111, in service_create",
2019-08-21 18:07:34 |         "    systemctl.enable(service, now=True)",
2019-08-21 18:07:34 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 53, in enable",
2019-08-21 18:07:34 |         "    systemctl(cmd, log)",
2019-08-21 18:07:34 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 33, in systemctl",
2019-08-21 18:07:34 |         "    raise SystemctlException(str(err))",
2019-08-21 18:07:34 |         "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_memcached-0vb7evm1']' returned non-zero exit s
tatus 1.",
2019-08-21 18:07:34 |         "Command '['systemctl', 'enable', '--now', 'tripleo_memcached-0vb7evm1']' returned non-zero exit status 1."



2019-08-21 18:07:34 | controller-0               : ok=217  changed=94   unreachable=0    failed=1    skipped=514  rescued=0    ignored=3
2019-08-21 18:07:34 | controller-1               : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2019-08-21 18:07:34 | controller-2               : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0




2019-08-21 18:11:29 |         "2019-08-21 18:10:59,827 ERROR: 59693 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', 'container-puppet-neutron', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,neutron_config,ovn_metadata_agent_config', '--env', 'NAME=neutron', '--env', 'HOSTNAME=compute-0', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--log-driver', 'json-file', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpweypvo26:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-neutron.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--volume', '/lib/modules:/lib/modules:ro', '--volume', '/run/openvswitch:/run/openvswitch:shared,z', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', '192.168.24.1:8787/rhosp15/openstack-neutron-server:20190819.1'] run failed after + mkdir -p /etc/puppet",
2019-08-21 18:11:29 |         "+ '[' -n file,file_line,concat,augeas,cron,neutron_config,ovn_metadata_agent_config ']'",
2019-08-21 18:11:29 |         "+ TAGS='--tags file,file_line,concat,augeas,cron,neutron_config,ovn_metadata_agent_config'",
2019-08-21 18:11:29 |         "+ origin_of_time=/var/lib/config-data/neutron.origin_of_time",
2019-08-21 18:11:29 |         "+ touch /var/lib/config-data/neutron.origin_of_time",
2019-08-21 18:11:29 |         "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,neutron_config,ovn_metadata_agent_config /etc/config.pp",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user:    (file: /etc/puppet/hiera.yaml)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: Undefined variable '::deploy_config_name'; \\n   (file & line not available)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n   (file & line not available)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)",
2019-08-21 18:11:29 |         "<13>Aug 21 18:10:55 puppet-user: Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/neutron/manifests/init.pp\", 470]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/neutron/ovn_metadata.pp\", 40]",


2019-08-21 18:11:29 | PLAY RECAP *********************************************************************
2019-08-21 18:11:29 | compute-0                  : ok=95   changed=34   unreachable=0    failed=1    skipped=560  rescued=0    ignored=2




2019-08-21 18:14:26 |         "Executing: systemctl enable --now tripleo_logrotate_crond-tquxr4a6",
2019-08-21 18:14:26 |         "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_logrotate_crond-tquxr4a6.service → /etc/systemd/system/tripleo_logrotate_crond-tquxr4a6.se
rvice.",
2019-08-21 18:14:26 |         "Job for tripleo_logrotate_crond-tquxr4a6.service failed because the service did not take the steps required by its unit configuration.",
2019-08-21 18:14:26 |         "See \"systemctl status tripleo_logrotate_crond-tquxr4a6.service\" and \"journalctl -xe\" for details.",
2019-08-21 18:14:26 |         "systemctl failed",
2019-08-21 18:14:26 |         "Traceback (most recent call last):",
2019-08-21 18:14:26 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 31, in systemctl",
2019-08-21 18:14:26 |         "    subprocess.check_call(cmd)",
2019-08-21 18:14:26 |         "  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call",
2019-08-21 18:14:26 |         "    raise CalledProcessError(retcode, cmd)",
2019-08-21 18:14:26 |         "subprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_logrotate_crond-tquxr4a6']' returned non-zero exit status 1.",
2019-08-21 18:14:26 |         "During handling of the above exception, another exception occurred:",
2019-08-21 18:14:26 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 111, in service_create",
2019-08-21 18:14:26 |         "    systemctl.enable(service, now=True)",
2019-08-21 18:14:26 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 53, in enable",
2019-08-21 18:14:26 |         "    systemctl(cmd, log)",
2019-08-21 18:14:26 |         "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 33, in systemctl",
2019-08-21 18:14:26 |         "    raise SystemctlException(str(err))",
2019-08-21 18:14:26 |         "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_logrotate_crond-tquxr4a6']' returned non-zero exit status 1.",
2019-08-21 18:14:26 |         "Command '['systemctl', 'enable', '--now', 'tripleo_logrotate_crond-tquxr4a6']' returned non-zero exit status 1.",


2019-08-21 18:14:26 | PLAY RECAP *********************************************************************
2019-08-21 18:14:26 | ceph-0                     : ok=104  changed=38   unreachable=0    failed=1    skipped=666  rescued=0    ignored=1
2019-08-21 18:14:26 | ceph-1                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2019-08-21 18:14:26 | ceph-2                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


Version-Release number of selected component (if applicable):

RHOS_TRUNK-15.0-RHEL-8-20190819.n.1

ansible-role-tripleo-modify-image.noarch      1.1.1-0.20190711170427.eabaed0.el8ost                    @rhelosp-15.0
ansible-tripleo-ipsec.noarch                  9.1.1-0.20190513190404.ffe104c.el8ost                    @rhelosp-15.0
openstack-heat-agents.noarch                  1.8.1-0.20190523210450.1e15344.el8ost                    @rhelosp-15.0
openstack-heat-api.noarch                     1:12.0.1-0.20190805120452.3476f1d.el8ost                 @rhelosp-15.0
openstack-heat-common.noarch                  1:12.0.1-0.20190805120452.3476f1d.el8ost                 @rhelosp-15.0
openstack-heat-engine.noarch                  1:12.0.1-0.20190805120452.3476f1d.el8ost                 @rhelosp-15.0
openstack-heat-monolith.noarch                1:12.0.1-0.20190805120452.3476f1d.el8ost                 @rhelosp-15.0
openstack-selinux.noarch                      0.8.19-0.20190813150447.72046d3.el8ost                   @rhelosp-15.0
openstack-tripleo-common.noarch               10.8.1-0.20190813170455.913b8de.el8ost                   @rhelosp-15.0
openstack-tripleo-common-containers.noarch    10.8.1-0.20190813170455.913b8de.el8ost                   @rhelosp-15.0
openstack-tripleo-heat-templates.noarch       10.6.1-0.20190819180520.6a38682.el8ost                   @rhelosp-15.0
openstack-tripleo-image-elements.noarch       10.4.1-0.20190705161217.2c8a6a5.el8ost                   @rhelosp-15.0
openstack-tripleo-puppet-elements.noarch      10.3.2-0.20190710165331.c89fe3c.el8ost                   @rhelosp-15.0
openstack-tripleo-validations.noarch          10.5.1-0.20190813200546.6f827f5.el8ost                   @rhelosp-15.0
puppet-openstack_extras.noarch                14.4.1-0.20190420090934.6b1b687.el8ost                   @rhelosp-15.0
puppet-openstacklib.noarch                    14.4.1-0.20190420125152.3719ca1.el8ost                   @rhelosp-15.0
puppet-tripleo.noarch                         10.5.1-0.20190812120435.ed6c6b0.el8ost                   @rhelosp-15.0
python-openstackclient-lang.noarch            3.18.0-0.20190312140834.6868499.el8ost                   @rhelosp-15.0
python3-openstackclient.noarch                3.18.0-0.20190312140834.6868499.el8ost                   @rhelosp-15.0
python3-openstacksdk.noarch                   0.27.0-0.20190405091843.4174082.el8ost                   @rhelosp-15.0
python3-tripleo-common.noarch                 10.8.1-0.20190813170455.913b8de.el8ost                   @rhelosp-15.0
python3-tripleoclient.noarch                  11.5.1-0.20190819174843.f83a1ed.el8ost                   @rhelosp-15.0
python3-tripleoclient-heat-installer.noarch   11.5.1-0.20190819174843.f83a1ed.el8ost                   @rhelosp-15.0


How reproducible:

consistent


Steps to Reproduce:
1. run openstack overcloud minor update 
2.
3.

Actual results:

RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/f351b7d0-c01c-468b-856c-748ca31f578c/ansible.log




Expected results:


Additional info: