Description of problem: OSP 9 to OSP 10 undercloud upgrade getting failed with following error messages: ~~~ 2016-12-20 14:47:19 - ESC[1;31mError: /usr/bin/nova-manage db sync returned 1 instead of one of [0]ESC[0m 2016-12-20 14:47:19 - ESC[1;31mError: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: /usr/bin/nova-manage db sync returned 1 instead of one of [0]ESC[0m 2016-12-20 14:47:19 - ESC[mNotice: /Stage[main]/Nova::Deps/Anchor[nova::dbsync::end]: Dependency Exec[nova-db-sync] has failures: true ESC[0m ========== Logs trauncate for the sake of brevity ======= + rc=6 + set -e + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2016-12-20 14:51:44,303] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] [2016-12-20 14:51:44,303] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1206, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 996, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 519, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 ~~~ Version-Release number of selected component (if applicable): RHEL OSP 10 How reproducible: Everytime for Cu. Steps to Reproduce: 1. Follow the official doc [a] steps and upgrade is getting failed. [a] https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/upgrading-red-hat-openstack-platform/#sect-Major-Updating_Director_Packages 2. 3. Actual results: Upgrade of undercloud is getting failed. Expected results: Upgrade should get completed successfully. Additional info: packages present on undercloud node. ~~~ # awk '/instack/ {print $1}' installed-rpms instack-5.0.0-1.el7ost.noarch instack-undercloud-5.1.0-4.el7ost.noarch # awk '/director/ {print $1}' installed-rpms rhosp-director-images-10.0-20161212.1.el7ost.noarch rhosp-director-images-9.0-20160818.1.el7ost.noarch rhosp-director-images-9.0-20161031.1.el7ost.noarch rhosp-director-images-ipa-10.0-20161212.1.el7ost.noarch rhosp-director-images-ipa-9.0-20160818.1.el7ost.noarch rhosp-director-images-ipa-9.0-20161031.1.el7ost.noarch ~~~
Tried to run it manually it got failed with following error : ~~~ $ dib-run-parts /usr/libexec/os-refresh-config/configure.d dib-run-parts Tue Dec 20 16:03:35 CET 2016 Running /usr/libexec/os-refresh-config/configure.d/20-os-apply-config [2016/12/20 04:03:35 PM] [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json [2016/12/20 04:03:35 PM] [ERROR] [Errno 13] Permission denied: '/usr/libexec/os-apply-config/templates/etc/os-collect-config.conf' [2016/12/20 04:03:35 PM] [ERROR] ('could not render moustache template /usr/libexec/os-apply-config/templates/etc/os-collect-config.conf', '/usr/libexec/os-apply-config/templates/etc/os-collect-config.conf') ~~~
So this BZ is actually covering two separate issues. The first issue is regarding nova db sync. As suggested by Vikrant, this can be worked around by running echo "delete from compute_nodes where uuid is NULL;" |mysql nova The second issue is regarding the heat DB and that the sync times out if the database is very large. As it is likely to be if one hasn't cleaned it up after deploying the errata that made the purge of deleted templates to work. The cleanup procedure is described in https://access.redhat.com/solutions/2573081