Bug 1658326
| Summary: | Major upgrade OSP13->OSP14 fails "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Mehdi ABAAKOUK <mabaakou> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 14.0 (Rocky) | CC: | apannu, ccamacho, dbecker, gfidente, mabaakou, mburns, mmagr, morazi, pkilambi, sasha, shdunne, yprokule | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | 14.0 (Rocky) | Flags: | mabaakou:
needinfo-
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-9.0.1-0.20181013060905.el7ost | Doc Type: | No Doc Update | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-01-11 11:55:18 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1642114 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
The issue reproduces.
There are also 2 restarting containers:
[root@overcloud-controller-0 ~]# docker ps|grep -i restart
5fb72b2734aa 192.168.24.1:8787/rhosp13/openstack-glance-api:13.0-65 "kolla_start" 5 hours ago Restarting (1) About an hour ago glance_api
19a9aaa85c90 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server:13.0-62 "kolla_start" 5 hours ago Restarting (1) About an hour ago swift_proxy
[root@overcloud-controller-0 ~]# docker logs glance_api
Traceback (most recent call last):
File "/usr/bin/glance-api", line 10, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main
config.parse_args()
File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args
default_config_files=default_config_files)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__
raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/glance/glance-api.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-api.conf
INFO:__main__:Deleting /etc/glance/glance-cache.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glance-cache.conf
INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Deleting /etc/ceph/rbdmap
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/rbdmap to /etc/ceph/rbdmap
INFO:__main__:Deleting /etc/ceph/ceph.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf
INFO:__main__:Deleting /etc/ceph/ceph.client.admin.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.admin.keyring to /etc/ceph/ceph.client.admin.keyring
INFO:__main__:Deleting /etc/ceph/ceph.mon.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.mon.keyring to /etc/ceph/ceph.mon.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.openstack.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/ceph.client.openstack.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.manila.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph.client.manila.keyring
INFO:__main__:Deleting /etc/ceph/ceph.client.radosgw.keyring
INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.radosgw.keyring to /etc/ceph/ceph.client.radosgw.keyring
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/lib/glance
INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring
++ cat /run_command
+ CMD='/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ [[ ! -d /var/log/kolla/glance ]]
+++ stat -c %a /var/log/kolla/glance
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/glance
++ . /usr/local/bin/kolla_glance_extend_start
+++ [[ -n '' ]]
++++ whoami
+++ [[ glance == \r\o\o\t ]]
+ echo 'Running command: '\''/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''
+ exec /usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf
Running command: '/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
Traceback (most recent call last):
File "/usr/bin/glance-api", line 10, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main
config.parse_args()
File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args
default_config_files=default_config_files)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__
raise ConfigFilesNotFoundError(self._namespace._files_not_found)
oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf
[root@overcloud-controller-0 ~]# docker logs swift_proxy
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/swift/account-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account-server.conf to /etc/swift/account-server.conf
INFO:__main__:Deleting /etc/swift/container-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container-server.conf to /etc/swift/container-server.conf
INFO:__main__:Deleting /etc/swift/keymaster.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/keymaster.conf to /etc/swift/keymaster.conf
INFO:__main__:Deleting /etc/swift/object-expirer.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-expirer.conf to /etc/swift/object-expirer.conf
INFO:__main__:Deleting /etc/swift/object-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-server.conf to /etc/swift/object-server.conf
INFO:__main__:Deleting /etc/swift/proxy-server.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/proxy-server.conf to /etc/swift/proxy-server.conf
INFO:__main__:Deleting /etc/swift/swift.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf
INFO:__main__:Deleting /etc/swift/account.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.ring.gz to /etc/swift/account.ring.gz
INFO:__main__:Deleting /etc/swift/container.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.ring.gz to /etc/swift/container.ring.gz
INFO:__main__:Deleting /etc/swift/object.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.ring.gz to /etc/swift/object.ring.gz
INFO:__main__:Deleting /etc/swift/account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.builder to /etc/swift/account.builder
INFO:__main__:Deleting /etc/swift/container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.builder to /etc/swift/container.builder
INFO:__main__:Deleting /etc/swift/object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.builder to /etc/swift/object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.account.builder to /etc/swift/backups/1544562933.account.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.container.builder to /etc/swift/backups/1544562933.container.builder
INFO:__main__:Deleting /etc/swift/backups/1544562933.object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.object.builder to /etc/swift/backups/1544562933.object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.account.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.builder to /etc/swift/backups/1544562939.account.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.account.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.ring.gz to /etc/swift/backups/1544562939.account.ring.gz
INFO:__main__:Deleting /etc/swift/backups/1544562939.object.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.builder to /etc/swift/backups/1544562939.object.builder
INFO:__main__:Deleting /etc/swift/backups/1544562939.object.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.ring.gz to /etc/swift/backups/1544562939.object.ring.gz
INFO:__main__:Deleting /etc/swift/backups/1544562940.container.builder
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.builder to /etc/swift/backups/1544562940.container.builder
INFO:__main__:Deleting /etc/swift/backups/1544562940.container.ring.gz
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.ring.gz to /etc/swift/backups/1544562940.container.ring.gz
INFO:__main__:Deleting /etc/rsyncd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD='/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+++ whoami
++ [[ swift == \r\o\o\t ]]
+ echo 'Running command: '\''/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'\'''
+ exec /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
Running command: '/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'
Traceback (most recent call last):
File "/usr/bin/swift-proxy-server", line 23, in <module>
sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 914, in run_wsgi
loadapp(conf_path, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 390, in loadapp
ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 374, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
object_type, name=name, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 450, in get_context
global_additions=global_additions)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
for name in pipeline[:-1]]
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
object_type, name=name, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
section)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
object_type, name=use, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context
object_type, name=name, global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
object_type, name=name)
File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 659, in find_egg_entry_point
for prot in protocol_options] or '(no entry points)'))))
LookupError: Entry point 's3api' not found in egg 'swift' (dir: /usr/lib/python2.7/site-packages; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: )
Created attachment 1513686 [details]
upgrade process output
Hi, I finished to reproduce/check this BZ, so quick question, what I did was to install OSP13 checking that I was using the default CeilometerAgentCentral heat resource, then when I ran the upgrade I included in all the upgrade steps a custom -e with the template including the ceilometer fix (at the end of each command so it should take precedence), and I didn't hit the issue. Can this be considered as a workaround in order to make this BZ not a blocker? At least not blocking RC and deliver the fix for RC? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Major upgrade OSP13->OSP14 fails "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database" Environment: openstack-ceilometer-notification-11.0.1-0.20181011093803.018ff32.el7ost.noarch openstack-ceilometer-common-11.0.1-0.20181011093803.018ff32.el7ost.noarch puppet-ironic-13.3.1-0.20181013115250.el7ost.noarch openstack-ironic-inspector-8.0.1-0.20180924215820.e89450c.el7ost.noarch openstack-ceilometer-central-11.0.1-0.20181011093803.018ff32.el7ost.noarch openstack-ceilometer-polling-11.0.1-0.20181011093803.018ff32.el7ost.noarch python-ironic-lib-2.14.0-0.20180810074837.344161b.el7ost.noarch openstack-ironic-conductor-11.1.1-0.20181012152841.el7ost.noarch openstack-ceilometer-compute-11.0.1-0.20181011093803.018ff32.el7ost.noarch puppet-ceilometer-13.3.1-0.20181013125758.4f7b0bf.el7ost.noarch python2-ironicclient-2.5.0-0.20180810135843.fb94fb8.el7ost.noarch openstack-ironic-common-11.1.1-0.20181012152841.el7ost.noarch openstack-ironic-api-11.1.1-0.20181012152841.el7ost.noarch openstack-ceilometer-ipmi-11.0.1-0.20181011093803.018ff32.el7ost.noarch python2-ceilometermiddleware-1.3.0-0.20180822210914.8332b9e.el7ost.noarch python2-ceilometerclient-2.9.1-0.20180817101107.3ff6bac.el7ost.noarch python-ceilometer-11.0.1-0.20181011093803.018ff32.el7ost.noarch python2-ironic-inspector-client-3.3.0-0.20180810080932.53bf4e8.el7ost.noarch instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch Steps to reproduce: 1. Deploy OSP13 with ironic in OC using ironic composable role: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \ -e /home/stack/templates/ironic.yaml \ -e /home/stack/templates/flat_networks.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \ -e /home/stack/docker-images.yaml ############################################################################### - name: IronicConductor description: | Ironic Conductor node role HostnameFormatDefault: '%stackname%-ironic-%index%' networks: - InternalApi ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::Collectd - OS::TripleO::Services::Docker - OS::TripleO::Services::Fluentd - OS::TripleO::Services::Ipsec - OS::TripleO::Services::IronicConductor - OS::TripleO::Services::IronicPxe - OS::TripleO::Services::Kernel - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::Rhsm - OS::TripleO::Services::SensuClient - OS::TripleO::Services::Snmp - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned parameter_defaults: ControllerCount: '3' OvercloudControlFlavor: 'controller' ComputeCount: '2' OvercloudComputeFlavor: 'compute' NtpServer: ["clock.redhat.com","clock2.redhat.com"] CephStorageCount: 3 OvercloudCephStorageFlavor: ceph IronicConductorCount: 2 OvercloudIronicConductorFlavor: ironrole 2) Try to upgrade the setup to OSP14. Result: The upgrade fails during overcloud upgrade phase 705b95ea8aed 192.168.24.1:8787/rhosp13/openstack-ceilometer-central:13.0-60 "/usr/bin/bootstra..." 10 hours ago Exited (1) 10 hours ago ceilometer_gnocchi_upgrade [root@overcloud-controller-0 ~]# docker logs ceilometer_gnocchi_upgrade usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH] [--debug] [--log-config-append PATH] [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR] [--log-file PATH] [--nodebug] [--noskip-gnocchi-resource-types] [--nouse-journal] [--nouse-json] [--nouse-syslog] [--nowatch-log-file] [--retry RETRY] [--skip-gnocchi-resource-types] [--syslog-log-facility SYSLOG_LOG_FACILITY] [--use-journal] [--use-json] [--use-syslog] [--version] [--watch-log-file] ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database