Bug 1642114

Summary: OSP13 Overcloud deploy fails on ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database
Product: Red Hat OpenStack Reporter: Pavel Sedlák <psedlak>
Component: openstack-tripleo-heat-templatesAssignee: Mehdi ABAAKOUK <mabaakou>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: bsachana, jschluet, mabaakou, mburns, pkilambi, sputhenp
Target Milestone: z3Keywords: Reopened, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.7-4.el7ost Doc Type: No Doc Update
Doc Text:
-
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-03 15:44:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1658326    

Description Pavel Sedlák 2018-10-23 15:59:59 UTC 
When running 'openstack overcloud deploy' it fails after ~50 minutes on overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:

>  Stack overcloud CREATE_FAILED 
> 
> overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
>   resource_type: OS::Heat::StructuredDeployment
>   physical_resource_id: 8297651f-c5b2-4092-89a5-b7e850fff6d9
>   status: CREATE_FAILED
>   status_reason: |
>     Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
>   deploy_stdout: |
>     ...
>             "                          [--version] [--watch-log-file]", 
>             "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database", 
>             "usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]"
>         ]
>     }
>     	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/4f8eb0bd-37cb-4401-8cdc-f5fbac424f6e_playbook.retry
>     
>     PLAY RECAP *********************************************************************
>     localhost                  : ok=6    changed=2    unreachable=0    failed=1   

Seems that --skip-metering-database parameter (and corresponding functionality for metering-database upgrading) was removed in https://github.com/openstack/ceilometer/commit/9323f07f977f320882f8b536c3b54835274826fc#diff-50d93c6e42b174e6332d1c217215f6dc

although it's still being used by THT https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-central.yaml#L138  (both stable/queens... and even in current master branch).

Version-Release number of selected component (if applicable):
From Undercloud:
> puppet-ceilometer.noarch             12.5.1-0.20180831170823.fbdfae9.el7ost @rhelosp-13.0-puddle
> openstack-tripleo-common.noarch      8.6.6-2.el7ost     @rhelosp-13.0-puddle
> openstack-tripleo-common-containers.noarch 8.6.6-2.el7ost     @rhelosp-13.0-puddle
> openstack-tripleo-heat-templates.noarch 8.0.7-3.el7ost     @rhelosp-13.0-puddle
> openstack-tripleo-image-elements.noarch 8.0.1-1.el7ost     @rhelosp-13.0-puddle
> openstack-tripleo-puppet-elements.noarch 8.0.1-1.el7ost     @rhelosp-13.0-puddle

From Overcloud controller-0:
> python-ceilometer.noarch            1:10.0.1-2.el7ost  @rhos-13.0-signed
> python2-ceilometerclient.noarch     2.9.0-0.20180210123537.3b4e35a.el7ost @rhos-13.0-signed
> python2-ceilometermiddleware.noarch 1.2.0-1.el7ost     @rhos-13.0-signed
> puppet-ceilometer.noarch            12.5.1-0.20180831170823.fbdfae9.el7ost @rhos-13.0-signed
> puppet-tripleo.noarch               8.3.6-2.el7ost     @rhos-13.0-signed
> openstack-ceilometer-central.noarch 1:10.0.1-2.el7ost  @rhos-13.0-signed
> openstack-ceilometer-common.noarch  1:10.0.1-2.el7ost  @rhos-13.0-signed
> openstack-ceilometer-compute.noarch 1:10.0.1-2.el7ost  @rhos-13.0-signed
> openstack-ceilometer-ipmi.noarch    1:10.0.1-2.el7ost  @rhos-13.0-signed
> openstack-ceilometer-notification.noarch 1:10.0.1-2.el7ost  @rhos-13.0-signed
> openstack-ceilometer-polling.noarch 1:10.0.1-2.el7ost  @rhos-13.0-signed
Comment 7 Sasha Smolyak 2018-11-01 12:13:59 UTC 
Deployment successful
Verified
Comment 11 errata-xmlrpc 2018-11-13 22:31:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587
Comment 12 Sasha Smolyak 2018-12-12 15:30:14 UTC 
Deploying 2018-12-07.1 puddle 
controller:1,compute:1,ceph:1,telemetry:1
Composable roles: telemetry.
Got an error while installing undercloud:

    2018-12-12 14:36:51.034 | TASK [print the last 60 lines of the undercloud installation] ******************
    2018-12-12 14:36:51.036 | task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-undercloud/deploy.yml:27
    2018-12-12 14:36:51.038 | changed: [undercloud-0] => {
    2018-12-12 14:36:51.040 |     "changed": true,
    2018-12-12 14:36:51.042 |     "cmd": [
    2018-12-12 14:36:51.045 |         "tail",
    2018-12-12 14:36:51.047 |         "-n60",
    2018-12-12 14:36:51.049 |         "undercloud_install.log"
    2018-12-12 14:36:51.052 |     ],
    2018-12-12 14:36:51.054 |     "delta": "0:00:00.003064",
    2018-12-12 14:36:51.056 |     "end": "2018-12-12 09:36:50.908156",
    2018-12-12 14:36:51.058 |     "rc": 0,
    2018-12-12 14:36:51.060 |     "start": "2018-12-12 09:36:50.905092"
    2018-12-12 14:36:51.062 | }
    2018-12-12 14:36:51.065 |
    2018-12-12 14:36:51.067 | STDOUT:
    2018-12-12 14:36:51.069 |
    2018-12-12 14:36:51.071 | 2018-12-12 09:36:35,679 INFO:    Rabbitmq plugin: 2.33
    2018-12-12 14:36:51.075 | 2018-12-12 09:36:35,680 INFO:    Neutron config: 2.40
    2018-12-12 14:36:51.077 | 2018-12-12 09:36:35,680 INFO:    Keystone config: 2.46
    2018-12-12 14:36:51.079 | 2018-12-12 09:36:35,680 INFO:       Heat config: 2.81
    2018-12-12 14:36:51.081 | 2018-12-12 09:36:35,680 INFO:     Keystone role: 2.94
    2018-12-12 14:36:51.084 | 2018-12-12 09:36:35,680 INFO:        Mysql user: 2.99
    2018-12-12 14:36:51.086 | 2018-12-12 09:36:35,680 INFO:    Config retrieval: 24.32
    2018-12-12 14:36:51.088 | 2018-12-12 09:36:35,680 INFO:      Nova cell v2: 3.34
    2018-12-12 14:36:51.090 | 2018-12-12 09:36:35,680 INFO:    Glance api config: 3.42
    2018-12-12 14:36:51.093 | 2018-12-12 09:36:35,680 INFO:     Keystone user: 38.38
    2018-12-12 14:36:51.095 | 2018-12-12 09:36:35,680 INFO:    Keystone user role: 48.52
    2018-12-12 14:36:51.097 | 2018-12-12 09:36:35,681 INFO:           Service: 49.45
    2018-12-12 14:36:51.099 | 2018-12-12 09:36:35,681 INFO:              Exec: 50.74
    2018-12-12 14:36:51.101 | 2018-12-12 09:36:35,681 INFO:    Keystone endpoint: 54.39
    2018-12-12 14:36:51.103 | 2018-12-12 09:36:35,681 INFO:     Ironic config: 6.02
    2018-12-12 14:36:51.106 | 2018-12-12 09:36:35,681 INFO:           Package: 631.35
    2018-12-12 14:36:51.108 | 2018-12-12 09:36:35,681 INFO:        Filebucket: 0.00
    2018-12-12 14:36:51.110 | 2018-12-12 09:36:35,681 INFO:     Mysql datadir: 9.48
    2018-12-12 14:36:51.113 | 2018-12-12 09:36:35,681 INFO: Version:
    2018-12-12 14:36:51.115 | 2018-12-12 09:36:35,681 INFO:            Config: 1544624209
    2018-12-12 14:36:51.117 | 2018-12-12 09:36:35,681 INFO:            Puppet: 4.8.2
    2018-12-12 14:36:51.120 | 2018-12-12 09:36:50,172 INFO: + rc=6
    2018-12-12 14:36:51.122 | 2018-12-12 09:36:50,172 INFO: + set -e
    2018-12-12 14:36:51.124 | 2018-12-12 09:36:50,172 INFO: + echo 'puppet apply exited with exit code 6'
    2018-12-12 14:36:51.126 | 2018-12-12 09:36:50,173 INFO: puppet apply exited with exit code 6
    2018-12-12 14:36:51.128 | 2018-12-12 09:36:50,173 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
    2018-12-12 14:36:51.131 | 2018-12-12 09:36:50,173 INFO: + exit 6
    2018-12-12 14:36:51.133 | 2018-12-12 09:36:50,173 INFO: [2018-12-12 09:36:50,173] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
    2018-12-12 14:36:51.135 | 2018-12-12 09:36:50,173 INFO:
    2018-12-12 14:36:51.137 | 2018-12-12 09:36:50,174 INFO: [2018-12-12 09:36:50,173] (os-refresh-config) [ERROR] Aborting...
    2018-12-12 14:36:51.139 | 2018-12-12 09:36:50,183 DEBUG: An exception occurred
    2018-12-12 14:36:51.142 | Traceback (most recent call last):
    2018-12-12 14:36:51.144 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2332, in install
    2018-12-12 14:36:51.146 |     _run_orc(instack_env)
    2018-12-12 14:36:51.148 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1599, in _run_orc
    2018-12-12 14:36:51.151 |     _run_live_command(args, instack_env, 'os-refresh-config')
    2018-12-12 14:36:51.153 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 668, in _run_live_command
    2018-12-12 14:36:51.685 |     raise RuntimeError('%s failed. See log for details.' % name)
    2018-12-12 14:36:51.687 | RuntimeError: os-refresh-config failed. See log for details.
    2018-12-12 14:36:51.689 | 2018-12-12 09:36:50,184 ERROR:
    2018-12-12 14:36:51.691 | #############################################################################
    2018-12-12 14:36:51.693 | Undercloud install failed.
    2018-12-12 14:36:51.695 |
    2018-12-12 14:36:51.697 | Reason: os-refresh-config failed. See log for details.
    2018-12-12 14:36:51.699 |
    2018-12-12 14:36:51.701 | See the previous output for details about what went wrong.  The full install
    2018-12-12 14:36:51.703 | log can be found at /home/stack/.instack/install-undercloud.log.
    2018-12-12 14:36:51.705 |
    2018-12-12 14:36:51.707 | #############################################################################
    2018-12-12 14:36:51.709 |
    2018-12-12 14:36:51.711 | Traceback (most recent call last):
    2018-12-12 14:36:51.713 |   File "<string>", line 1, in <module>
    2018-12-12 14:36:51.717 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2332, in install
    2018-12-12 14:36:51.719 |     _run_orc(instack_env)
    2018-12-12 14:36:51.721 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1599, in _run_orc
    2018-12-12 14:36:51.724 |     _run_live_command(args, instack_env, 'os-refresh-config')
    2018-12-12 14:36:51.726 |   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 668, in _run_live_command
    2018-12-12 14:36:51.728 |     raise RuntimeError('%s failed. See log for details.' % name)
    2018-12-12 14:36:51.730 | RuntimeError: os-refresh-config failed. See log for details.
    2018-12-12 14:36:51.732 | Command 'instack-install-undercloud' returned non-zero exit status 1
    2018-12-12 14:36:51.734 |
    2018-12-12 14:36:51.736 |
    2018-12-12 14:36:51.738 | TASK [print any nested error entries in undercloud installation] ***************
    2018-12-12 14:36:51.740 | task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-undercloud/deploy.yml:30
    2018-12-12 14:36:51.742 | changed: [undercloud-0] => {
    2018-12-12 14:36:51.744 |     "changed": true,
    2018-12-12 14:36:51.746 |     "cmd": "grep -B2 -A5 -i 'info:.*error:' undercloud_install.log || true",
    2018-12-12 14:36:51.748 |     "delta": "0:00:00.006566",
    2018-12-12 14:36:51.750 |     "end": "2018-12-12 09:36:51.481334",
    2018-12-12 14:36:51.752 |     "rc": 0,
    2018-12-12 14:36:51.754 |     "start": "2018-12-12 09:36:51.474768"
    2018-12-12 14:36:51.756 | }
    2018-12-12 14:36:51.758 |
    2018-12-12 14:36:51.760 | STDOUT:
    2018-12-12 14:36:51.762 |
    2018-12-12 14:36:51.764 | 2018-12-12 09:36:13,513 INFO: Notice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns:                           [--use-journal] [--use-json] [--use-syslog]
    2018-12-12 14:36:51.766 | 2018-12-12 09:36:13,513 INFO: Notice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns:                           [--version] [--watch-log-file]
    2018-12-12 14:36:51.768 | 2018-12-12 09:36:13,513 INFO: Notice: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database
    2018-12-12 14:36:51.770 | 2018-12-12 09:36:13,513 INFO: Error: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]
    2018-12-12 14:36:51.772 | 2018-12-12 09:36:13,513 INFO: Error: /Stage[main]/Main/Exec[ceilo-gnocchi-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 2 instead of one of [0]
    2018-12-12 14:36:51.774 | 2018-12-12 09:36:17,137 INFO: Notice: /Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure: ensure changed 'stopped' to 'running'
    2018-12-12 14:36:51.776 | 2018-12-12 09:36:17,138 INFO: Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::end]: Triggered 'refresh' from 4 events
    2018-12-12 14:36:51.778 | 2018-12-12 09:36:17,144 INFO: Notice: /Stage[main]/Nova::Logging/File[/var/log/nova/nova-manage.log]/seluser: seluser changed 'unconfined_u' to 'system_u'
    2018-12-12 14:36:51.780 | 2018-12-12 09:36:20,610 INFO: Notice: /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]: Triggered 'refresh' from 2 events
    2018-12-12 14:36:51.782 | 2018-12-12 09:36:21,099 INFO: Notice: /Stage[main]/Swift::Storage::Account/Swift::Service[swift-account-reaper]/Service[swift-account-reaper]/ensure: ensure changed 'stopped' to 'running'
    2018-12-12 14:36:51.784 |
    2018-12-12 14:36:51.786 |
    2018-12-12 14:36:51.788 | TASK [fail] ********************************************************************
    2018-12-12 14:36:51.790 | task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-undercloud/deploy.yml:33
    2018-12-12 14:36:51.792 | fatal: [undercloud-0]: FAILED! => {
    2018-12-12 14:36:51.794 |     "changed": false
    2018-12-12 14:36:51.796 | }
    2018-12-12 14:36:51.798 |
    2018-12-12 14:36:51.801 | MSG:
    2018-12-12 14:36:51.803 |
    2018-12-12 14:36:51.805 | Undercloud deployment failed... :(


Reopening the bug
Comment 13 Mehdi ABAAKOUK 2019-01-03 08:33:10 UTC 
This is not the same bug, the original was a tripleo heat templates bug. 

The new one is an instack-undercloud bug.

Please fill free to open another bug.