Major upgrade OSP13->OSP14 fails "ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database" Environment: openstack-ceilometer-notification-11.0.1-0.20181011093803.018ff32.el7ost.noarch openstack-ceilometer-common-11.0.1-0.20181011093803.018ff32.el7ost.noarch puppet-ironic-13.3.1-0.20181013115250.el7ost.noarch openstack-ironic-inspector-8.0.1-0.20180924215820.e89450c.el7ost.noarch openstack-ceilometer-central-11.0.1-0.20181011093803.018ff32.el7ost.noarch openstack-ceilometer-polling-11.0.1-0.20181011093803.018ff32.el7ost.noarch python-ironic-lib-2.14.0-0.20180810074837.344161b.el7ost.noarch openstack-ironic-conductor-11.1.1-0.20181012152841.el7ost.noarch openstack-ceilometer-compute-11.0.1-0.20181011093803.018ff32.el7ost.noarch puppet-ceilometer-13.3.1-0.20181013125758.4f7b0bf.el7ost.noarch python2-ironicclient-2.5.0-0.20180810135843.fb94fb8.el7ost.noarch openstack-ironic-common-11.1.1-0.20181012152841.el7ost.noarch openstack-ironic-api-11.1.1-0.20181012152841.el7ost.noarch openstack-ceilometer-ipmi-11.0.1-0.20181011093803.018ff32.el7ost.noarch python2-ceilometermiddleware-1.3.0-0.20180822210914.8332b9e.el7ost.noarch python2-ceilometerclient-2.9.1-0.20180817101107.3ff6bac.el7ost.noarch python-ceilometer-11.0.1-0.20181011093803.018ff32.el7ost.noarch python2-ironic-inspector-client-3.3.0-0.20180810080932.53bf4e8.el7ost.noarch instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch Steps to reproduce: 1. Deploy OSP13 with ironic in OC using ironic composable role: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \ -e /home/stack/templates/ironic.yaml \ -e /home/stack/templates/flat_networks.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \ -e /home/stack/docker-images.yaml ############################################################################### - name: IronicConductor description: | Ironic Conductor node role HostnameFormatDefault: '%stackname%-ironic-%index%' networks: - InternalApi ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::Collectd - OS::TripleO::Services::Docker - OS::TripleO::Services::Fluentd - OS::TripleO::Services::Ipsec - OS::TripleO::Services::IronicConductor - OS::TripleO::Services::IronicPxe - OS::TripleO::Services::Kernel - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::Rhsm - OS::TripleO::Services::SensuClient - OS::TripleO::Services::Snmp - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned parameter_defaults: ControllerCount: '3' OvercloudControlFlavor: 'controller' ComputeCount: '2' OvercloudComputeFlavor: 'compute' NtpServer: ["clock.redhat.com","clock2.redhat.com"] CephStorageCount: 3 OvercloudCephStorageFlavor: ceph IronicConductorCount: 2 OvercloudIronicConductorFlavor: ironrole 2) Try to upgrade the setup to OSP14. Result: The upgrade fails during overcloud upgrade phase 705b95ea8aed 192.168.24.1:8787/rhosp13/openstack-ceilometer-central:13.0-60 "/usr/bin/bootstra..." 10 hours ago Exited (1) 10 hours ago ceilometer_gnocchi_upgrade [root@overcloud-controller-0 ~]# docker logs ceilometer_gnocchi_upgrade usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH] [--debug] [--log-config-append PATH] [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR] [--log-file PATH] [--nodebug] [--noskip-gnocchi-resource-types] [--nouse-journal] [--nouse-json] [--nouse-syslog] [--nowatch-log-file] [--retry RETRY] [--skip-gnocchi-resource-types] [--syslog-log-facility SYSLOG_LOG_FACILITY] [--use-journal] [--use-json] [--use-syslog] [--version] [--watch-log-file] ceilometer-upgrade: error: unrecognized arguments: --skip-metering-database
The issue reproduces. There are also 2 restarting containers: [root@overcloud-controller-0 ~]# docker ps|grep -i restart 5fb72b2734aa 192.168.24.1:8787/rhosp13/openstack-glance-api:13.0-65 "kolla_start" 5 hours ago Restarting (1) About an hour ago glance_api 19a9aaa85c90 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server:13.0-62 "kolla_start" 5 hours ago Restarting (1) About an hour ago swift_proxy [root@overcloud-controller-0 ~]# docker logs glance_api Traceback (most recent call last): File "/usr/bin/glance-api", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main config.parse_args() File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args default_config_files=default_config_files) File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/glance/glance-api.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-api.conf INFO:__main__:Deleting /etc/glance/glance-cache.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glance-cache.conf INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf INFO:__main__:Deleting /etc/ceph/rbdmap INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/rbdmap to /etc/ceph/rbdmap INFO:__main__:Deleting /etc/ceph/ceph.conf INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf INFO:__main__:Deleting /etc/ceph/ceph.client.admin.keyring INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.admin.keyring to /etc/ceph/ceph.client.admin.keyring INFO:__main__:Deleting /etc/ceph/ceph.mon.keyring INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.mon.keyring to /etc/ceph/ceph.mon.keyring INFO:__main__:Deleting /etc/ceph/ceph.client.openstack.keyring INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/ceph.client.openstack.keyring INFO:__main__:Deleting /etc/ceph/ceph.client.manila.keyring INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph.client.manila.keyring INFO:__main__:Deleting /etc/ceph/ceph.client.radosgw.keyring INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.radosgw.keyring to /etc/ceph/ceph.client.radosgw.keyring INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/lib/glance INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring ++ cat /run_command + CMD='/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -d /var/log/kolla/glance ]] +++ stat -c %a /var/log/kolla/glance ++ [[ 2755 != \7\5\5 ]] ++ chmod 755 /var/log/kolla/glance ++ . /usr/local/bin/kolla_glance_extend_start +++ [[ -n '' ]] ++++ whoami +++ [[ glance == \r\o\o\t ]] + echo 'Running command: '\''/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\''' + exec /usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf Running command: '/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf' Traceback (most recent call last): File "/usr/bin/glance-api", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/glance/cmd/api.py", line 78, in main config.parse_args() File "/usr/lib/python2.7/site-packages/glance/common/config.py", line 756, in parse_args default_config_files=default_config_files) File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2504, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/glance/glance-image-import.conf [root@overcloud-controller-0 ~]# docker logs swift_proxy + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/swift/account-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account-server.conf to /etc/swift/account-server.conf INFO:__main__:Deleting /etc/swift/container-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container-server.conf to /etc/swift/container-server.conf INFO:__main__:Deleting /etc/swift/keymaster.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/keymaster.conf to /etc/swift/keymaster.conf INFO:__main__:Deleting /etc/swift/object-expirer.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-expirer.conf to /etc/swift/object-expirer.conf INFO:__main__:Deleting /etc/swift/object-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-server.conf to /etc/swift/object-server.conf INFO:__main__:Deleting /etc/swift/proxy-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/proxy-server.conf to /etc/swift/proxy-server.conf INFO:__main__:Deleting /etc/swift/swift.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf INFO:__main__:Deleting /etc/swift/account.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.ring.gz to /etc/swift/account.ring.gz INFO:__main__:Deleting /etc/swift/container.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.ring.gz to /etc/swift/container.ring.gz INFO:__main__:Deleting /etc/swift/object.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.ring.gz to /etc/swift/object.ring.gz INFO:__main__:Deleting /etc/swift/account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.builder to /etc/swift/account.builder INFO:__main__:Deleting /etc/swift/container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.builder to /etc/swift/container.builder INFO:__main__:Deleting /etc/swift/object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.builder to /etc/swift/object.builder INFO:__main__:Deleting /etc/swift/backups/1544562933.account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.account.builder to /etc/swift/backups/1544562933.account.builder INFO:__main__:Deleting /etc/swift/backups/1544562933.container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.container.builder to /etc/swift/backups/1544562933.container.builder INFO:__main__:Deleting /etc/swift/backups/1544562933.object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562933.object.builder to /etc/swift/backups/1544562933.object.builder INFO:__main__:Deleting /etc/swift/backups/1544562939.account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.builder to /etc/swift/backups/1544562939.account.builder INFO:__main__:Deleting /etc/swift/backups/1544562939.account.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.account.ring.gz to /etc/swift/backups/1544562939.account.ring.gz INFO:__main__:Deleting /etc/swift/backups/1544562939.object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.builder to /etc/swift/backups/1544562939.object.builder INFO:__main__:Deleting /etc/swift/backups/1544562939.object.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562939.object.ring.gz to /etc/swift/backups/1544562939.object.ring.gz INFO:__main__:Deleting /etc/swift/backups/1544562940.container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.builder to /etc/swift/backups/1544562940.container.builder INFO:__main__:Deleting /etc/swift/backups/1544562940.container.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1544562940.container.ring.gz to /etc/swift/backups/1544562940.container.ring.gz INFO:__main__:Deleting /etc/rsyncd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf INFO:__main__:Writing out command to execute ++ cat /run_command + CMD='/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start +++ whoami ++ [[ swift == \r\o\o\t ]] + echo 'Running command: '\''/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'\''' + exec /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf Running command: '/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf' Traceback (most recent call last): File "/usr/bin/swift-proxy-server", line 23, in <module> sys.exit(run_wsgi(conf_file, 'proxy-server', **options)) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 914, in run_wsgi loadapp(conf_path, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 390, in loadapp ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 374, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 450, in get_context global_additions=global_additions) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context for name in pipeline[:-1]] File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context section) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use object_type, name=use, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 66, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg return loader.get_context(object_type, name, global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context object_type, name=name) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 659, in find_egg_entry_point for prot in protocol_options] or '(no entry points)')))) LookupError: Entry point 's3api' not found in egg 'swift' (dir: /usr/lib/python2.7/site-packages; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: )
Created attachment 1513686 [details] upgrade process output
Hi, I finished to reproduce/check this BZ, so quick question, what I did was to install OSP13 checking that I was using the default CeilometerAgentCentral heat resource, then when I ran the upgrade I included in all the upgrade steps a custom -e with the template including the ceilometer fix (at the end of each command so it should take precedence), and I didn't hit the issue. Can this be considered as a workaround in order to make this BZ not a blocker? At least not blocking RC and deliver the fix for RC?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045