Bug 1658326

Summary: Major upgrade OSP13->OSP14 fails "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database"
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Mehdi ABAAKOUK <mabaakou>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: apannu, ccamacho, dbecker, gfidente, mabaakou, mburns, mmagr, morazi, pkilambi, sasha, shdunne, yprokule
Target Milestone: rcKeywords: Triaged
Target Release: 14.0 (Rocky)Flags: mabaakou: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060905.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:55:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1642114    
Bug Blocks:    
Attachments:
Description Flags
upgrade process output none

Description Alexander Chuzhoy 2018-12-11 18:42:42 UTC
Major upgrade OSP13->OSP14 fails "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database"


Environment:
openstack-ceilometer-notification-11.0.1-0.20181011093803.018ff32.el7ost.noarch
openstack-ceilometer-common-11.0.1-0.20181011093803.018ff32.el7ost.noarch
puppet-ironic-13.3.1-0.20181013115250.el7ost.noarch
openstack-ironic-inspector-8.0.1-0.20180924215820.e89450c.el7ost.noarch
openstack-ceilometer-central-11.0.1-0.20181011093803.018ff32.el7ost.noarch
openstack-ceilometer-polling-11.0.1-0.20181011093803.018ff32.el7ost.noarch
python-ironic-lib-2.14.0-0.20180810074837.344161b.el7ost.noarch
openstack-ironic-conductor-11.1.1-0.20181012152841.el7ost.noarch
openstack-ceilometer-compute-11.0.1-0.20181011093803.018ff32.el7ost.noarch
puppet-ceilometer-13.3.1-0.20181013125758.4f7b0bf.el7ost.noarch
python2-ironicclient-2.5.0-0.20180810135843.fb94fb8.el7ost.noarch
openstack-ironic-common-11.1.1-0.20181012152841.el7ost.noarch
openstack-ironic-api-11.1.1-0.20181012152841.el7ost.noarch
openstack-ceilometer-ipmi-11.0.1-0.20181011093803.018ff32.el7ost.noarch
python2-ceilometermiddleware-1.3.0-0.20180822210914.8332b9e.el7ost.noarch
python2-ceilometerclient-2.9.1-0.20180817101107.3ff6bac.el7ost.noarch
python-ceilometer-11.0.1-0.20181011093803.018ff32.el7ost.noarch
python2-ironic-inspector-client-3.3.0-0.20180810080932.53bf4e8.el7ost.noarch
instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch
openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch


Steps to reproduce:
1. Deploy OSP13 with ironic in OC using ironic composable role:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /home/stack/templates/nodes_data.yaml \
-r /home/stack/roles_data.yaml \
-e  /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \
-e /home/stack/templates/ironic.yaml \
-e /home/stack/templates/flat_networks.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \
-e /home/stack/docker-images.yaml 


###############################################################################
- name: IronicConductor
  description: |
    Ironic Conductor node role
  HostnameFormatDefault: '%stackname%-ironic-%index%'
  networks:
    - InternalApi
  ServicesDefault:
    - OS::TripleO::Services::Aide
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Fluentd
    - OS::TripleO::Services::Ipsec
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::IronicPxe
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::LoginDefs
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::Rhsm
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned


parameter_defaults:
    ControllerCount: '3'
    OvercloudControlFlavor: 'controller'
    ComputeCount: '2'
    OvercloudComputeFlavor: 'compute'
    NtpServer: ["clock.redhat.com","clock2.redhat.com"]
    CephStorageCount: 3
    OvercloudCephStorageFlavor: ceph
    IronicConductorCount: 2
    OvercloudIronicConductorFlavor: ironrole


2)
Try to upgrade the setup to OSP14.


Result:

The upgrade fails during overcloud upgrade phase



705b95ea8aed        192.168.24.1:8787/rhosp13/openstack-ceilometer-central:13.0-60          "/usr/bin/bootstra..."   10 hours ago        Exited (1) 10 hours ago                          ceilometer_gnocchi_upgrade



[root@overcloud-controller-0 ~]# docker logs ceilometer_gnocchi_upgrade
usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]
                          [--debug] [--log-config-append PATH]
                          [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]
                          [--log-file PATH] [--nodebug]
                          [--noskip-gnocchi-resource-types] [--nouse-journal]
                          [--nouse-json] [--nouse-syslog] [--nowatch-log-file]
                          [--retry RETRY] [--skip-gnocchi-resource-types]
                          [--syslog-log-facility SYSLOG_LOG_FACILITY]
                          [--use-journal] [--use-json] [--use-syslog]
                          [--version] [--watch-log-file]
ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database

Comment 2 Alexander Chuzhoy 2018-12-12 06:31:14 UTC
The issue reproduces.

There are also 2 restarting containers:
[root@overcloud-controller-0 ~]# docker ps|grep -i restart
5fb72b2734aa        192.168.24.1:8787/rhosp13/openstack-glance-api:13.0-65                  "kolla_start"            5 hours ago         Restarting (1) About an hour ago                       glance_api
19a9aaa85c90        192.168.24.1:8787/rhosp13/openstack-swift-proxy-server:13.0-62          "kolla_start"            5 hours ago         Restarting (1) About an hour ago                       swift_proxy




[root@overcloud-controller-0 ~]# docker logs glance_api
Traceback (most recent call last):
  File "/usr/bin/glance-api", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main
    config.parse_args()
  File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args
    default_config_files=default_config_files)
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__
    raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/glance/glance-api.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-api.conf
INFO:__main__:Deleting /etc/glance/glance-cache.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glance-cache.conf
INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Deleting /etc/ceph/rbdmap
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/rbdmap to /etc/ceph/rbdmap
INFO:__main__:Deleting /etc/ceph/ceph.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf
INFO:__main__:Deleting /etc/ceph/ceph.client.admin.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.admin.keyring to /etc/ceph/ceph.client.admin.keyring
INFO:__main__:Deleting /etc/ceph/ceph.mon.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.mon.keyring to /etc/ceph/ceph.mon.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.openstack.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/ceph.client.openstack.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.manila.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph.client.manila.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.radosgw.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.radosgw.keyring to /etc/ceph/ceph.client.radosgw.keyring
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/lib/glance
INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring
++ cat /run_command
+ CMD='/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ [[ ! -d /var/log/kolla/glance ]]
+++ stat -c %a /var/log/kolla/glance
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/glance
++ . /usr/local/bin/kolla_glance_extend_start
+++ [[ -n '' ]]
++++ whoami
+++ [[ glance == \r\o\o\t ]]
+ echo 'Running command: '\''/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''
+ exec /usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf
Running command: '/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
Traceback (most recent call last):
  File "/usr/bin/glance-api", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main
    config.parse_args()
  File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args
    default_config_files=default_config_files)
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__
    raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf



[root@overcloud-controller-0 ~]# docker logs swift_proxy




+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/swift/account-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account-server.conf to /etc/swift/account-server.conf
INFO:__main__:Deleting /etc/swift/container-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container-server.conf to /etc/swift/container-server.conf
INFO:__main__:Deleting /etc/swift/keymaster.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/keymaster.conf to /etc/swift/keymaster.conf
INFO:__main__:Deleting /etc/swift/object-expirer.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-expirer.conf to /etc/swift/object-expirer.conf
INFO:__main__:Deleting /etc/swift/object-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-server.conf to /etc/swift/object-server.conf
INFO:__main__:Deleting /etc/swift/proxy-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/proxy-server.conf to /etc/swift/proxy-server.conf
INFO:__main__:Deleting /etc/swift/swift.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf
INFO:__main__:Deleting /etc/swift/account.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.ring.gz to /etc/swift/account.ring.gz
INFO:__main__:Deleting /etc/swift/container.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.ring.gz to /etc/swift/container.ring.gz
INFO:__main__:Deleting /etc/swift/object.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.ring.gz to /etc/swift/object.ring.gz
INFO:__main__:Deleting /etc/swift/account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.builder to /etc/swift/account.builder
INFO:__main__:Deleting /etc/swift/container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.builder to /etc/swift/container.builder
INFO:__main__:Deleting /etc/swift/object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.builder to /etc/swift/object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.account.builder to /etc/swift/backups/1544562933.account.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.container.builder to /etc/swift/backups/1544562933.container.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.object.builder to /etc/swift/backups/1544562933.object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.builder to /etc/swift/backups/1544562939.account.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.account.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.ring.gz to /etc/swift/backups/1544562939.account.ring.gz
INFO:__main__:Deleting /etc/swift/backups/1544562939.object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.builder to /etc/swift/backups/1544562939.object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.object.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.ring.gz to /etc/swift/backups/1544562939.object.ring.gz
INFO:__main__:Deleting /etc/swift/backups/1544562940.container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.builder to /etc/swift/backups/1544562940.container.builder
INFO:__main__:Deleting /etc/swift/backups/1544562940.container.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.ring.gz to /etc/swift/backups/1544562940.container.ring.gz
INFO:__main__:Deleting /etc/rsyncd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD='/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+++ whoami
++ [[ swift == \r\o\o\t ]]
+ echo 'Running command: '\''/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'\'''
+ exec /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
Running command: '/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'
Traceback (most recent call last):
  File "/usr/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 914, in run_wsgi
    loadapp(conf_path, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 390, in loadapp
    ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 374, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 450, in get_context
    global_additions=global_additions)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
    for name in pipeline[:-1]]
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 659, in find_egg_entry_point
    for prot in protocol_options] or '(no entry points)'))))
LookupError: Entry point 's3api' not found in egg 'swift' (dir: /usr/lib/python2.7/site-packages; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: )

Comment 5 Alexander Chuzhoy 2018-12-12 14:23:40 UTC
Created attachment 1513686 [details]
upgrade process output

Comment 9 Carlos Camacho 2018-12-14 10:39:40 UTC
Hi,

I finished to reproduce/check this BZ, so quick question,
what I did was to install OSP13 checking that I was using the default CeilometerAgentCentral heat resource,
then when I ran the upgrade I included in all the upgrade steps a custom -e with the template including the ceilometer fix
(at the end of each command so it should take precedence), and I didn't hit the issue.

Can this be considered as a workaround in order to make this BZ not a blocker?

At least not blocking RC and deliver the fix for RC?

Comment 21 errata-xmlrpc 2019-01-11 11:55:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045