Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1418782

Summary: rhosp-director: Undercloud deployment fails with: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: puppet-tripleoAssignee: Angus Thomas <athomas>
Status: CLOSED CURRENTRELEASE QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: urgent    
Version: 11.0 (Ocata)CC: aschultz, dbecker, emacchi, jjoyce, jschluet, mburns, mcornea, morazi, ohochman, rhel-osp-director-maint, slinaber, tvignaud, yprokule
Target Milestone: ga   
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-28 20:45:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
install-undercloud.log
none
sosreport from the undercloud node none

Description Alexander Chuzhoy 2017-02-02 17:21:58 UTC 
rhosp-director:  Undercloud deployment fails with: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout

Environment:
instack-undercloud-6.0.0-0.20170127055514.317db76.el7ost.noarch

Steps to reproduce:
Attempt to deploy undercloud with "openstack undercloud install"


Result:
Fails:
2017-02-02 11:36:05,996 INFO: Error: Command exceeded timeout
2017-02-02 11:36:05,997 INFO: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout

...
...

2017-02-02 11:50:28,241 INFO: Warning: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]: Skipping because of failed dependencies
2017-02-02 11:50:31,680 INFO: Notice: Applied catalog in 2802.33 seconds
2017-02-02 11:50:48,870 INFO: + rc=6
2017-02-02 11:50:48,870 INFO: + set -e
2017-02-02 11:50:48,871 INFO: + echo 'puppet apply exited with exit code 6'
2017-02-02 11:50:48,872 INFO: puppet apply exited with exit code 6
2017-02-02 11:50:48,872 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-02-02 11:50:48,873 INFO: + exit 6
2017-02-02 11:50:48,874 INFO: [2017-02-02 11:50:48,871] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-02-02 11:50:48,874 INFO: 
2017-02-02 11:50:48,875 INFO: [2017-02-02 11:50:48,873] (os-refresh-config) [ERROR] Aborting...
2017-02-02 11:50:48,883 ERROR: 
#############################################################################
Undercloud install failed.

Reason: os-refresh-config failed. See log for details.

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Expected result:
Successfully deployed undercloud.
Comment 1 Alexander Chuzhoy 2017-02-02 17:37:44 UTC 
Created attachment 1247162 [details]
install-undercloud.log
Comment 2 Alexander Chuzhoy 2017-02-02 17:38:57 UTC 
Created attachment 1247163 [details]
sosreport from the undercloud node
Comment 3 Alexander Chuzhoy 2017-02-06 18:14:13 UTC 
Was able to workaround the reported issue with:
sudo sed -i 's/^innodb_file_per_table.*//' /etc/my.cnf.d/galera.cnf
sudo systemctl restart mariadb

prior to running "openstack undercloud install"
Comment 4 Alex Schultz 2017-02-06 18:17:05 UTC 
In reply to #3, that is only if you previously had run the install and did not  have the following two patches https://review.openstack.org/#/c/428435/ and https://review.openstack.org/#/c/428843/ applied.  Under normal cases, that doesn't fix the problem due to the setting being reapplied when the install process is run.
Comment 5 Omri Hochman 2017-02-13 14:25:44 UTC 
reproduced on my bare-metal environment
Comment 6 Alex Schultz 2017-02-13 16:34:35 UTC 
Omri, please provide software versions, logs, and hardware specs. If possible, providing access to the environment would also suffice.
Comment 7 Omri Hochman 2017-02-14 03:08:06 UTC 
hi Alex - my issue got resolve using the workaround :
sudo tee -a /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template <<'EOF'
# https://bugzilla.redhat.com/show_bug.cgi?id=1418782
# NOTE(aschultz): raise upper limit on nova DB syncs for undercloud only.
# There is no way this should take 15 minutes and if it does we now have way
# different problems. But rather than block undercloud installs let's increase
# the timeout for these actions. See LP#1661396 for more details.
nova::db::sync::db_sync_timeout: 900
nova::db::sync_api::db_sync_timeout: 900
EOF