Description of problem: Deployment fails with error: (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] controller:3,compute:2,ceph:3 Telemetry enabled in undercloud Templates: custom_templates: parameter_defaults: GnocchiBackend: file Version-Release number of selected component (if applicable): Puddle 2019-01-03.2 How reproducible: 100% Steps to Reproduce: 1. Try to deploy with above setup 2. 3. Actual results: Deployment fails Expected results: Deployment passes Additional info: From installation log: 2019-01-06 15:41:30.169 | TASK [install the undercloud] ************************************************** 2019-01-06 15:41:30.171 | task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-undercloud/deploy.yml:22 2019-01-06 15:41:30.173 | fatal: [undercloud-0]: FAILED! => { 2019-01-06 15:41:30.175 | "changed": true, 2019-01-06 15:41:30.177 | "cmd": [ 2019-01-06 15:41:30.179 | "bash", 2019-01-06 15:41:30.181 | "~/undercloud_deploy.sh" 2019-01-06 15:41:30.183 | ], 2019-01-06 15:41:30.185 | "delta": "0:17:44.014207", 2019-01-06 15:41:30.187 | "end": "2019-01-06 10:41:29.933998", 2019-01-06 15:41:30.188 | "rc": 1, 2019-01-06 15:41:30.191 | "start": "2019-01-06 10:23:45.919791" 2019-01-06 15:41:30.192 | } 2019-01-06 15:41:30.194 | 2019-01-06 15:41:30.196 | MSG: 2019-01-06 15:41:30.198 | 2019-01-06 15:41:30.200 | non-zero return code 2019-01-06 15:41:30.203 | 2019-01-06 15:41:30.205 | ...ignoring 2019-01-06 15:41:30.206 | 2019-01-06 15:41:30.470 | TASK [print the last 60 lines of the undercloud installation] ****************** 2019-01-06 15:41:30.472 | task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-undercloud/deploy.yml:27 2019-01-06 15:41:30.474 | changed: [undercloud-0] => { 2019-01-06 15:41:30.476 | "changed": true, 2019-01-06 15:41:30.478 | "cmd": [ 2019-01-06 15:41:30.480 | "tail", 2019-01-06 15:41:30.482 | "-n60", 2019-01-06 15:41:30.484 | "undercloud_install.log" 2019-01-06 15:41:30.486 | ], 2019-01-06 15:41:30.488 | "delta": "0:00:00.003759", 2019-01-06 15:41:30.490 | "end": "2019-01-06 10:41:30.424066", 2019-01-06 15:41:30.492 | "rc": 0, 2019-01-06 15:41:30.494 | "start": "2019-01-06 10:41:30.420307" 2019-01-06 15:41:30.496 | } 2019-01-06 15:41:30.498 | 2019-01-06 15:41:30.500 | STDOUT: 2019-01-06 15:41:30.502 | 2019-01-06 15:41:30.504 | 2019-01-06 10:41:12,404 INFO: Heat config: 2.67 2019-01-06 15:41:30.506 | 2019-01-06 10:41:12,404 INFO: File: 2.72 2019-01-06 15:41:30.508 | 2019-01-06 10:41:12,404 INFO: Neutron config: 2.82 2019-01-06 15:41:30.510 | 2019-01-06 10:41:12,405 INFO: Keystone role: 3.00 2019-01-06 15:41:30.512 | 2019-01-06 10:41:12,405 INFO: Config retrieval: 24.05 2019-01-06 15:41:30.514 | 2019-01-06 10:41:12,405 INFO: Nova cell v2: 3.07 2019-01-06 15:41:30.516 | 2019-01-06 10:41:12,405 INFO: Keystone config: 3.53 2019-01-06 15:41:30.518 | 2019-01-06 10:41:12,405 INFO: Glance api config: 3.57 2019-01-06 15:41:30.520 | 2019-01-06 10:41:12,405 INFO: Keystone user: 38.67 2019-01-06 15:41:30.522 | 2019-01-06 10:41:12,405 INFO: Package: 434.47 2019-01-06 15:41:30.524 | 2019-01-06 10:41:12,405 INFO: Keystone user role: 48.92 2019-01-06 15:41:30.526 | 2019-01-06 10:41:12,405 INFO: Service: 50.35 2019-01-06 15:41:30.528 | 2019-01-06 10:41:12,405 INFO: Exec: 51.74 2019-01-06 15:41:30.530 | 2019-01-06 10:41:12,405 INFO: Keystone endpoint: 55.65 2019-01-06 15:41:30.532 | 2019-01-06 10:41:12,405 INFO: Ironic config: 6.82 2019-01-06 15:41:30.534 | 2019-01-06 10:41:12,406 INFO: Mysql datadir: 9.50 2019-01-06 15:41:30.536 | 2019-01-06 10:41:12,406 INFO: Filebucket: 0.00 2019-01-06 15:41:30.538 | 2019-01-06 10:41:12,406 INFO: Total: 935.90 2019-01-06 15:41:30.539 | 2019-01-06 10:41:12,406 INFO: Version: 2019-01-06 15:41:30.542 | 2019-01-06 10:41:12,406 INFO: Config: 1546788267 2019-01-06 15:41:30.544 | 2019-01-06 10:41:12,406 INFO: Puppet: 4.8.2 2019-01-06 15:41:30.546 | 2019-01-06 10:41:29,831 INFO: + rc=6 2019-01-06 15:41:30.548 | 2019-01-06 10:41:29,831 INFO: + set -e 2019-01-06 15:41:30.550 | 2019-01-06 10:41:29,832 INFO: + echo 'puppet apply exited with exit code 6' 2019-01-06 15:41:30.552 | 2019-01-06 10:41:29,832 INFO: puppet apply exited with exit code 6 2019-01-06 15:41:30.554 | 2019-01-06 10:41:29,832 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']' 2019-01-06 15:41:30.556 | 2019-01-06 10:41:29,832 INFO: + exit 6 2019-01-06 15:41:30.558 | 2019-01-06 10:41:29,832 INFO: [2019-01-06 10:41:29,831] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] 2019-01-06 15:41:30.560 | 2019-01-06 10:41:29,832 INFO: 2019-01-06 15:41:30.562 | 2019-01-06 10:41:29,832 INFO: [2019-01-06 10:41:29,832] (os-refresh-config) [ERROR] Aborting... 2019-01-06 15:41:30.564 | 2019-01-06 10:41:29,841 DEBUG: An exception occurred 2019-01-06 15:41:30.566 | Traceback (most recent call last): 2019-01-06 15:41:30.568 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2332, in install 2019-01-06 15:41:30.570 | _run_orc(instack_env) 2019-01-06 15:41:30.571 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1599, in _run_orc 2019-01-06 15:41:30.573 | _run_live_command(args, instack_env, 'os-refresh-config') 2019-01-06 15:41:30.575 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 668, in _run_live_command 2019-01-06 15:41:30.577 | raise RuntimeError('%s failed. See log for details.' % name) 2019-01-06 15:41:30.579 | RuntimeError: os-refresh-config failed. See log for details. 2019-01-06 15:41:30.581 | 2019-01-06 10:41:29,842 ERROR: 2019-01-06 15:41:30.583 | ############################################################################# 2019-01-06 15:41:30.585 | Undercloud install failed. 2019-01-06 15:41:30.587 | 2019-01-06 15:41:30.589 | Reason: os-refresh-config failed. See log for details. 2019-01-06 15:41:30.591 | 2019-01-06 15:41:30.593 | See the previous output for details about what went wrong. The full install 2019-01-06 15:41:30.595 | log can be found at /home/stack/.instack/install-undercloud.log. 2019-01-06 15:41:30.597 | 2019-01-06 15:41:30.599 | ############################################################################# 2019-01-06 15:41:30.601 | 2019-01-06 15:41:30.603 | Traceback (most recent call last): 2019-01-06 15:41:30.605 | File "<string>", line 1, in <module> 2019-01-06 15:41:30.607 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2332, in install 2019-01-06 15:41:30.609 | _run_orc(instack_env) 2019-01-06 15:41:30.611 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1599, in _run_orc 2019-01-06 15:41:30.613 | _run_live_command(args, instack_env, 'os-refresh-config') 2019-01-06 15:41:30.615 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 668, in _run_live_command 2019-01-06 15:41:30.617 | raise RuntimeError('%s failed. See log for details.' % name) 2019-01-06 15:41:30.618 | RuntimeError: os-refresh-config failed. See log for details. 2019-01-06 15:41:30.620 | Command 'instack-install-undercloud' returned non-zero exit status 1
Do you have sosreport from the undercloud available?
Created attachment 1519159 [details] /home/stack from undercloud
sosreport from undercloud: https://drive.google.com/drive/folders/1YhQzmMWjhutx2sthWZzbsBSITNDV0O85?usp=sharing
*** Bug 1663261 has been marked as a duplicate of this bug. ***
Actual error is in ceilometer-upgrade 2019-01-08 04:58:25,543 INFO: ^[[1;31mError: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]^[[0m 2019-01-08 04:58:25,543 INFO: ^[[1;31mError: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]^[[0m
Specifically the ceilometer-upgrade doesn't have the --skip-metering-database option. 2019-01-08 04:58:25,541 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]^[[0m 2019-01-08 04:58:25,541 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--debug] [--log-config-append PATH]^[[0m 2019-01-08 04:58:25,541 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]^[[0m 2019-01-08 04:58:25,541 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--log-file PATH] [--nodebug]^[[0m 2019-01-08 04:58:25,541 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--noskip-gnocchi-resource-types] [--nouse-journal]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--nouse-json] [--nouse-syslog] [--nowatch-log-file]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--retry RETRY] [--skip-gnocchi-resource-types]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--syslog-log-facility SYSLOG_LOG_FACILITY]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--use-journal] [--use-json] [--use-syslog]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: [--version] [--watch-log-file]^[[0m 2019-01-08 04:58:25,542 INFO: ^[[mNotice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database^[[0m 2019-01-08 04:58:25,543 INFO: ^[[1;31mError: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]^[[0m 2019-01-08 04:58:25,543 INFO: ^[[1;31mError: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]^[[0m
*** This bug has been marked as a duplicate of bug 1659044 ***
*** Bug 1667424 has been marked as a duplicate of this bug. ***
Hello, DHS is currently hitting this bug causing the undercloud update failures.
The problem is caused by /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp file from instack-undercloud-8.4.6-4.el7ost.noarch package. Please note that during openstack undercloud install copy of this file is created in /etc/puppet/manifests/ and it is used during the next installation attempt. Resolution which works in my case: [root@director13 ~]# cat << EOF > /tmp/BZ1667424.patch --- /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp 2018-11-26 10:45:03.000000000 -0500 +++ /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp.new 2019-02-01 06:28:59.394972476 -0500 @@ -205,7 +205,7 @@ exec { 'ceilo-gnocchi-upgrade': - command => 'ceilometer-upgrade --skip-metering-database', + command => 'ceilometer-upgrade', path => ['/usr/bin', '/usr/sbin'], } EOF [root@director13 ~]# patch /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp < /tmp/BZ1667424.patch patching file /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp Hunk #1 succeeded at 207 with fuzz 2 (offset 2 lines). ### If you have already made openstack undercloud install attempt, you also need to patch copy of the file created in /etc/puppet/manifests/ [root@director13 ~]# patch /etc/puppet/manifests/puppet-stack-config.pp < /tmp/BZ1667424.patch patching file /etc/puppet/manifests/puppet-stack-config.pp Hunk #1 succeeded at 207 with fuzz 2 (offset 2 lines). Then you can re-run openstack undercloud install.
I've deployed openstack with the following: puddle : 2019-02-25.2 config: controller:3,compute:2,ceph:3 Telemetry enabled in undercloud Templates: custom_templates: parameter_defaults: GnocchiBackend: file No errors found during deployment. Deployment was successful!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0448