overcloud ironic service is missing db sync in fast_forward_upgrade_tasks Environment: openstack-ironic-conductor-6.2.4-3.el7ost.noarch puppet-ironic-9.5.0-3.el7ost.noarch python-ironicclient-1.7.1-1.el7ost.noarch python-ironic-lib-2.1.3-1.el7ost.noarch openstack-ironic-common-6.2.4-3.el7ost.noarch openstack-ironic-api-6.2.4-3.el7ost.noarch python-ironic-inspector-client-1.10.0-1.el7ost.noarch During FFU, specifically section 6.2 here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/fast_forward_upgrades/assembly-upgrading_the_overcloud#upgrading_all_controller_nodes the process fails with: "Error running [\'docker\', \'run\', \'--name\', \'ironic_db_sync\', \'--label\', \'config_id=tripleo_step3\', \'--label\', \'container_name=ironic_db_sync\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"start_order\\": 1, \\"image\\": \\"192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\\", \\"command\\": \\"/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c \\\\\'ironic-dbsync --config-file /etc/ironic/ironic.conf\\\\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\\", \\"/var/log/containers/ironic:/var/log/ironic\\", \\"/var/log/containers/httpd/ironic-api:/var/log/httpd\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}\', \'--net=host\', \'--privileged=false\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\', \'--volume=/var/log/containers/ironic:/var/log/ironic\', \'--volume=/var/log/containers/httpd/ironic-api:/var/log/httpd\', \'192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\', \'/usr/bin/bootstrap_host_exec\', \'ironic_api\', \'su\', \'ironic\', \'-s\', \'/bin/bash\', \'-c\', \\"\'ironic-dbsync\\", \'--config-file\', \\"/etc/ironic/ironic.conf\'\\"]. [1]", ', Basically, ironic has no fast forward upgrade tasks for the db sync.
*** Bug 1598542 has been marked as a duplicate of this bug. ***
With the patch applied, the upgrade fails running the following command: [root@controller-0 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade Traceback (most recent call last): File "/bin/ironic-dbsync", line 6, in <module> from ironic.cmd.dbsync import main File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 63, in <module> dbapi.backfill_version_column, File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 243, in __getattr__ self._load_backend() File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 238, in _load_backend backend_mod = importutils.import_module(backend_path) File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module __import__(import_str) File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 203, in <module> class Connection(api.Connection): File "/usr/lib/python2.7/site-packages/ironic/common/profiler.py", line 59, in decorator if CONF.profiler.enabled: File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2489, in __getattr__ raise NoSuchOptError(name) oslo_config.cfg.NoSuchOptError: no such option profiler in group [DEFAULT]
Can you check the versions of all ironic-related packages before and after the upgrade attempt: $ rpm -qa | grep ironic But I suspect that we need to make sure ironic pulls in the right version of osprofiler. Can you check the version of python2-osprofiler before and after upgrade? What I do not understand though is why you end up running ironic-dbsync outside of the container? The osprofiler support was added in Pike, and IIRC in Pike we should be containerized, no? Maybe this needs input from the upgrades team. Also please note that I'll be on PTO next week, so we'll need someone to take this over.
Tried manually to run update between osp11 and osp12 (this is where the problem starts). Checking yum history: Transaction performed with: Installed rpm-4.11.3-32.el7.x86_64 @anaconda/7.5 Installed subscription-manager-1.20.11-1.el7_5.x86_64 @koji-override-0/7.5 Installed yum-3.4.3-158.el7.noarch @anaconda/7.5 Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda/7.5 Packages Altered: Updated openstack-ironic-api-1:7.0.4-1.el7ost.noarch @rhelosp-11.0-puddle Update 1:9.1.4-1.el7ost.noarch @rhelosp-12.0-puddle Updated openstack-ironic-common-1:7.0.4-1.el7ost.noarch @rhelosp-11.0-puddle Update 1:9.1.4-1.el7ost.noarch @rhelosp-12.0-puddle Updated openstack-ironic-conductor-1:7.0.4-1.el7ost.noarch @rhelosp-11.0-puddle Update 1:9.1.4-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-amqp-1.4.6-1.el7ost.noarch @rhos-10.0-signed Update 2.1.4-2.el7ost.noarch @rhelosp-12.0-puddle Updated python-cinderclient-1.9.0-6.el7ost.noarch @rhos-10.0-signed Update 3.1.0-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-glanceclient-1:2.5.0-2.el7ost.noarch @rhos-10.0-signed Update 1:2.8.0-2.el7ost.noarch @rhelosp-12.0-puddle Updated python-kombu-1:3.0.32-2.el7ost.noarch @rhos-10.0-signed Update 1:4.0.2-5.el7ost.noarch @rhelosp-12.0-puddle Updated python-migrate-0.10.0-1.el7ost.noarch @rhos-10.0-signed Update 0.11.0-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-neutronclient-6.0.1-1.el7ost.noarch @rhos-10.0-signed Update 6.5.0-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-db-4.17.1-1.el7ost.noarch @rhelosp-11.0-puddle Update 4.25.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-db-lang-4.17.1-1.el7ost.noarch @rhelosp-11.0-puddle Update 4.25.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-messaging-5.17.3-1.el7ost.noarch @rhelosp-11.0-puddle Update 5.30.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-middleware-3.19.1-1.el7ost.noarch @rhos-10.0-signed Update 3.30.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-middleware-lang-3.19.1-1.el7ost.noarch @rhos-10.0-signed Update 3.30.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-policy-1.18.0-1.el7ost.noarch @rhelosp-11.0-puddle Update 1.25.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-policy-lang-1.18.0-1.el7ost.noarch @rhelosp-11.0-puddle Update 1.25.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-proliantutils-2.2.0-3.el7ost.noarch @rhos-10.0-signed Update 2.4.0-4.el7ost.noarch @rhelosp-12.0-puddle Dep-Install python-scciclient-0.5.0-1.el7ost.noarch @rhelosp-12.0-puddle Dep-Install python-sushy-1.1.0-1.el7ost.noarch @rhelosp-12.0-puddle Dep-Install python-vine-1.1.3-2.el7ost.noarch @rhelosp-12.0-puddle Updated python-webob-1.6.1-2.el7ost.noarch @rhos-10.0-signed Update 1.7.2-1.el7ost.noarch @rhelosp-12.0-puddle Dep-Install python2-statsd-3.2.1-5.el7ost.noarch @rhelosp-11.0-puddle Packages Altered: Updated python-oslo-cache-1.14.0-1.1.el7ost.noarch @rhos-10.0-signed Update 1.25.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-cache-lang-1.14.0-1.1.el7ost.noarch @rhos-10.0-signed Update 1.25.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-concurrency-3.14.1-1.el7ost.noarch @rhos-10.0-signed Update 3.21.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-concurrency-lang-3.14.1-1.el7ost.noarch @rhos-10.0-signed Update 3.21.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-privsep-1.13.2-1.el7ost.noarch @rhos-10.0-signed Update 1.22.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-privsep-lang-1.13.2-1.el7ost.noarch @rhos-10.0-signed Update 1.22.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-reports-1.14.0-1.el7ost.noarch @rhos-10.0-signed Update 1.22.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-rootwrap-5.1.2-1.el7ost.noarch @rhos-10.0-signed Update 5.9.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-service-1.16.1-1.el7ost.noarch @rhos-10.0-signed Update 1.25.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-sphinx-4.7.0-1.el7ost.noarch @rhos-10.0-signed Obsoleted python-oslo-sphinx-4.7.0-1.el7ost.noarch @rhos-10.0-signed Obsoleting python-oslo-sphinx-4.15.1-2.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-versionedobjects-1.17.1-1.el7ost.noarch @rhos-10.0-signed Update 1.26.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-versionedobjects-lang-1.17.1-1.el7ost.noarch @rhos-10.0-signed Update 1.26.2-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-vmware-2.14.1-1.el7ost.noarch @rhos-10.0-signed Update 2.23.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslo-vmware-lang-2.14.1-1.el7ost.noarch @rhos-10.0-signed Update 2.23.1-1.el7ost.noarch @rhelosp-12.0-puddle Updated python-oslotest-2.10.0-1.el7ost.noarch @rhos-10.0-signed Update 2.17.1-1.el7ost.noarch @rhelosp-12.0-puddle
If python-osprofiler is updated to a version from osp12 - the ironic-dbsync doesn't show the error: Dependencies Resolved ============================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================== Updating: python-osprofiler noarch 1.11.0-1.el7ost rhelosp-12.0-puddle 115 k Transaction Summary ============================================================================================================================================================================================== Upgrade 1 Package Total download size: 115 k Is this ok [y/d/N]: y Downloading packages: No Presto metadata available for rhelosp-12.0-puddle python-osprofiler-1.11.0-1.el7ost.noarch.rpm | 115 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : python-osprofiler-1.11.0-1.el7ost.noarch 1/2 Cleanup : python-osprofiler-1.4.0-1.el7ost.noarch 2/2 Verifying : python-osprofiler-1.11.0-1.el7ost.noarch 1/2 Verifying : python-osprofiler-1.4.0-1.el7ost.noarch 2/2 Updated: python-osprofiler.noarch 0:1.11.0-1.el7ost Complete! [root@controller-1 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade Option "rpc_backend" from group "DEFAULT" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade dbefd6bdaa2c -> 3d86a077a3f2, add port physical network INFO [alembic.runtime.migration] Running upgrade 3d86a077a3f2 -> 868cb606a74a, add version field in base class [root@controller-1 ~]# echo $? 0
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Environment: openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574