Bug 1418782 - rhosp-director: Undercloud deployment fails with: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout
Summary: rhosp-director: Undercloud deployment fails with: Error: /Stage[main]/Nova::...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ga
: 11.0 (Ocata)
Assignee: Angus Thomas
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-02 17:21 UTC by Alexander Chuzhoy
Modified: 2017-02-28 20:45 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-28 20:45:20 UTC
Target Upstream Version:


Attachments (Terms of Use)
install-undercloud.log (251.33 KB, text/plain)
2017-02-02 17:37 UTC, Alexander Chuzhoy
no flags Details
sosreport from the undercloud node (9.63 MB, application/x-xz)
2017-02-02 17:38 UTC, Alexander Chuzhoy
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1661396 0 None None None 2017-02-02 21:44:33 UTC
OpenStack gerrit 428435 0 None None None 2017-02-02 21:44:58 UTC
OpenStack gerrit 428843 0 None None None 2017-02-06 18:17:37 UTC

Description Alexander Chuzhoy 2017-02-02 17:21:58 UTC
rhosp-director:  Undercloud deployment fails with: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout

Environment:
instack-undercloud-6.0.0-0.20170127055514.317db76.el7ost.noarch

Steps to reproduce:
Attempt to deploy undercloud with "openstack undercloud install"


Result:
Fails:
2017-02-02 11:36:05,996 INFO: Error: Command exceeded timeout
2017-02-02 11:36:05,997 INFO: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout

...
...

2017-02-02 11:50:28,241 INFO: Warning: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]: Skipping because of failed dependencies
2017-02-02 11:50:31,680 INFO: Notice: Applied catalog in 2802.33 seconds
2017-02-02 11:50:48,870 INFO: + rc=6
2017-02-02 11:50:48,870 INFO: + set -e
2017-02-02 11:50:48,871 INFO: + echo 'puppet apply exited with exit code 6'
2017-02-02 11:50:48,872 INFO: puppet apply exited with exit code 6
2017-02-02 11:50:48,872 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-02-02 11:50:48,873 INFO: + exit 6
2017-02-02 11:50:48,874 INFO: [2017-02-02 11:50:48,871] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-02-02 11:50:48,874 INFO: 
2017-02-02 11:50:48,875 INFO: [2017-02-02 11:50:48,873] (os-refresh-config) [ERROR] Aborting...
2017-02-02 11:50:48,883 ERROR: 
#############################################################################
Undercloud install failed.

Reason: os-refresh-config failed. See log for details.

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Expected result:
Successfully deployed undercloud.

Comment 1 Alexander Chuzhoy 2017-02-02 17:37:44 UTC
Created attachment 1247162 [details]
install-undercloud.log

Comment 2 Alexander Chuzhoy 2017-02-02 17:38:57 UTC
Created attachment 1247163 [details]
sosreport from the undercloud node

Comment 3 Alexander Chuzhoy 2017-02-06 18:14:13 UTC
Was able to workaround the reported issue with:
sudo sed -i 's/^innodb_file_per_table.*//' /etc/my.cnf.d/galera.cnf
sudo systemctl restart mariadb

prior to running "openstack undercloud install"

Comment 4 Alex Schultz 2017-02-06 18:17:05 UTC
In reply to #3, that is only if you previously had run the install and did not  have the following two patches https://review.openstack.org/#/c/428435/ and https://review.openstack.org/#/c/428843/ applied.  Under normal cases, that doesn't fix the problem due to the setting being reapplied when the install process is run.

Comment 5 Omri Hochman 2017-02-13 14:25:44 UTC
reproduced on my bare-metal environment

Comment 6 Alex Schultz 2017-02-13 16:34:35 UTC
Omri, please provide software versions, logs, and hardware specs. If possible, providing access to the environment would also suffice.

Comment 7 Omri Hochman 2017-02-14 03:08:06 UTC
hi Alex - my issue got resolve using the workaround :
sudo tee -a /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template <<'EOF'
# https://bugzilla.redhat.com/show_bug.cgi?id=1418782
# NOTE(aschultz): raise upper limit on nova DB syncs for undercloud only.
# There is no way this should take 15 minutes and if it does we now have way
# different problems. But rather than block undercloud installs let's increase
# the timeout for these actions. See LP#1661396 for more details.
nova::db::sync::db_sync_timeout: 900
nova::db::sync_api::db_sync_timeout: 900
EOF


Note You need to log in before you can comment on or make changes to this bug.