Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1598191

Summary:	overcloud ironic service is missing db sync in fast_forward_upgrade_tasks
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	openstack-tripleo-heat-templates	Assignee:	Dmitry Tantsur <dtantsur>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	bfournie, dtantsur, jschluet, mburns, mcornea, sasha, srevivo
Target Milestone:	z2	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-8.0.4-3.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1599436 (view as bug list)		Environment:
Last Closed:	2018-08-29 16:37:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1599436

Description Alexander Chuzhoy 2018-07-04 14:59:45 UTC

overcloud ironic service is missing db sync in fast_forward_upgrade_tasks
Environment:
openstack-ironic-conductor-6.2.4-3.el7ost.noarch
puppet-ironic-9.5.0-3.el7ost.noarch
python-ironicclient-1.7.1-1.el7ost.noarch
python-ironic-lib-2.1.3-1.el7ost.noarch
openstack-ironic-common-6.2.4-3.el7ost.noarch
openstack-ironic-api-6.2.4-3.el7ost.noarch
python-ironic-inspector-client-1.10.0-1.el7ost.noarch



During FFU, specifically section 6.2 here:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/fast_forward_upgrades/assembly-upgrading_the_overcloud#upgrading_all_controller_nodes 

the process fails with:
"Error running [\'docker\', \'run\', \'--name\', \'ironic_db_sync\', \'--label\', \'config_id=tripleo_step3\', \'--label\', \'container_name=ironic_db_sync\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"start_order\\": 1, \\"image\\": \\"192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\\", \\"command\\": \\"/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c \\\\\'ironic-dbsync --config-file /etc/ironic/ironic.conf\\\\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\\", \\"/var/log/containers/ironic:/var/log/ironic\\", \\"/var/log/containers/httpd/ironic-api:/var/log/httpd\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}\', \'--net=host\', \'--privileged=false\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\', \'--volume=/var/log/containers/ironic:/var/log/ironic\', \'--volume=/var/log/containers/httpd/ironic-api:/var/log/httpd\', \'192.168.24.1:8787/rhosp13/openstack-ironic-api:2018-07-03.3\', \'/usr/bin/bootstrap_host_exec\', \'ironic_api\', \'su\', \'ironic\', \'-s\', \'/bin/bash\', \'-c\', \\"\'ironic-dbsync\\", \'--config-file\', \\"/etc/ironic/ironic.conf\'\\"]. [1]", ',


Basically,
ironic has no fast forward upgrade tasks for the db sync.

Comment 1 Alexander Chuzhoy 2018-07-05 18:28:35 UTC

*** Bug 1598542 has been marked as a duplicate of this bug. ***

Comment 2 Alexander Chuzhoy 2018-07-05 18:29:47 UTC

With the patch applied, the upgrade fails running the following command:

[root@controller-0 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade
Traceback (most recent call last):
  File "/bin/ironic-dbsync", line 6, in <module>
    from ironic.cmd.dbsync import main
  File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 63, in <module>
    dbapi.backfill_version_column,
  File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 243, in __getattr__
    self._load_backend()
  File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 238, in _load_backend
    backend_mod = importutils.import_module(backend_path)
  File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module
    __import__(import_str)
  File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 203, in <module>
    class Connection(api.Connection):
  File "/usr/lib/python2.7/site-packages/ironic/common/profiler.py", line 59, in decorator
    if CONF.profiler.enabled:
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2489, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option profiler in group [DEFAULT]

Comment 3 Dmitry Tantsur 2018-07-06 09:05:35 UTC

Can you check the versions of all ironic-related packages before and after the upgrade attempt:

 $ rpm -qa | grep ironic

But I suspect that we need to make sure ironic pulls in the right version of osprofiler. Can you check the version of python2-osprofiler before and after upgrade?

What I do not understand though is why you end up running ironic-dbsync outside of the container? The osprofiler support was added in Pike, and IIRC in Pike we should be containerized, no? Maybe this needs input from the upgrades team.

Also please note that I'll be on PTO next week, so we'll need someone to take this over.

Comment 4 Alexander Chuzhoy 2018-07-06 13:46:07 UTC

Tried manually to run update between osp11 and osp12 (this is where the problem starts).
Checking yum history:

Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                    @anaconda/7.5
    Installed     subscription-manager-1.20.11-1.el7_5.x86_64 @koji-override-0/7.5
    Installed     yum-3.4.3-158.el7.noarch                    @anaconda/7.5
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64     @anaconda/7.5
Packages Altered:
    Updated     openstack-ironic-api-1:7.0.4-1.el7ost.noarch       @rhelosp-11.0-puddle
    Update                           1:9.1.4-1.el7ost.noarch       @rhelosp-12.0-puddle
    Updated     openstack-ironic-common-1:7.0.4-1.el7ost.noarch    @rhelosp-11.0-puddle
    Update                              1:9.1.4-1.el7ost.noarch    @rhelosp-12.0-puddle
    Updated     openstack-ironic-conductor-1:7.0.4-1.el7ost.noarch @rhelosp-11.0-puddle
    Update                                 1:9.1.4-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated     python-amqp-1.4.6-1.el7ost.noarch                  @rhos-10.0-signed
    Update                  2.1.4-2.el7ost.noarch                  @rhelosp-12.0-puddle
    Updated     python-cinderclient-1.9.0-6.el7ost.noarch          @rhos-10.0-signed
    Update                          3.1.0-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated     python-glanceclient-1:2.5.0-2.el7ost.noarch        @rhos-10.0-signed
    Update                          1:2.8.0-2.el7ost.noarch        @rhelosp-12.0-puddle
    Updated     python-kombu-1:3.0.32-2.el7ost.noarch              @rhos-10.0-signed
    Update                   1:4.0.2-5.el7ost.noarch               @rhelosp-12.0-puddle
    Updated     python-migrate-0.10.0-1.el7ost.noarch              @rhos-10.0-signed
    Update                     0.11.0-1.el7ost.noarch              @rhelosp-12.0-puddle
    Updated     python-neutronclient-6.0.1-1.el7ost.noarch         @rhos-10.0-signed
    Update                           6.5.0-1.el7ost.noarch         @rhelosp-12.0-puddle
    Updated     python-oslo-db-4.17.1-1.el7ost.noarch              @rhelosp-11.0-puddle
    Update                     4.25.1-1.el7ost.noarch              @rhelosp-12.0-puddle
    Updated     python-oslo-db-lang-4.17.1-1.el7ost.noarch         @rhelosp-11.0-puddle
    Update                          4.25.1-1.el7ost.noarch         @rhelosp-12.0-puddle
    Updated     python-oslo-messaging-5.17.3-1.el7ost.noarch       @rhelosp-11.0-puddle
    Update                            5.30.2-1.el7ost.noarch       @rhelosp-12.0-puddle
    Updated     python-oslo-middleware-3.19.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                             3.30.1-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated     python-oslo-middleware-lang-3.19.1-1.el7ost.noarch @rhos-10.0-signed
    Update                                  3.30.1-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated     python-oslo-policy-1.18.0-1.el7ost.noarch          @rhelosp-11.0-puddle
    Update                         1.25.2-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated     python-oslo-policy-lang-1.18.0-1.el7ost.noarch     @rhelosp-11.0-puddle
    Update                              1.25.2-1.el7ost.noarch     @rhelosp-12.0-puddle
    Updated     python-proliantutils-2.2.0-3.el7ost.noarch         @rhos-10.0-signed
    Update                           2.4.0-4.el7ost.noarch         @rhelosp-12.0-puddle
    Dep-Install python-scciclient-0.5.0-1.el7ost.noarch            @rhelosp-12.0-puddle
    Dep-Install python-sushy-1.1.0-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Dep-Install python-vine-1.1.3-2.el7ost.noarch                  @rhelosp-12.0-puddle
    Updated     python-webob-1.6.1-2.el7ost.noarch                 @rhos-10.0-signed
    Update                   1.7.2-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Dep-Install python2-statsd-3.2.1-5.el7ost.noarch               @rhelosp-11.0-puddle





Packages Altered:
    Updated    python-oslo-cache-1.14.0-1.1.el7ost.noarch               @rhos-10.0-signed
    Update                       1.25.1-1.el7ost.noarch                 @rhelosp-12.0-puddle
    Updated    python-oslo-cache-lang-1.14.0-1.1.el7ost.noarch          @rhos-10.0-signed
    Update                            1.25.1-1.el7ost.noarch            @rhelosp-12.0-puddle
    Updated    python-oslo-concurrency-3.14.1-1.el7ost.noarch           @rhos-10.0-signed
    Update                             3.21.1-1.el7ost.noarch           @rhelosp-12.0-puddle
    Updated    python-oslo-concurrency-lang-3.14.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                                  3.21.1-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated    python-oslo-privsep-1.13.2-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.22.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-privsep-lang-1.13.2-1.el7ost.noarch          @rhos-10.0-signed
    Update                              1.22.1-1.el7ost.noarch          @rhelosp-12.0-puddle
    Updated    python-oslo-reports-1.14.0-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.22.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-rootwrap-5.1.2-1.el7ost.noarch               @rhos-10.0-signed
    Update                          5.9.2-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-service-1.16.1-1.el7ost.noarch               @rhos-10.0-signed
    Update                         1.25.1-1.el7ost.noarch               @rhelosp-12.0-puddle
    Updated    python-oslo-sphinx-4.7.0-1.el7ost.noarch                 @rhos-10.0-signed
    Obsoleted  python-oslo-sphinx-4.7.0-1.el7ost.noarch                 @rhos-10.0-signed
    Obsoleting python-oslo-sphinx-4.15.1-2.el7ost.noarch                @rhelosp-12.0-puddle
    Updated    python-oslo-versionedobjects-1.17.1-1.el7ost.noarch      @rhos-10.0-signed
    Update                                  1.26.2-1.el7ost.noarch      @rhelosp-12.0-puddle
    Updated    python-oslo-versionedobjects-lang-1.17.1-1.el7ost.noarch @rhos-10.0-signed
    Update                                       1.26.2-1.el7ost.noarch @rhelosp-12.0-puddle
    Updated    python-oslo-vmware-2.14.1-1.el7ost.noarch                @rhos-10.0-signed
    Update                        2.23.1-1.el7ost.noarch                @rhelosp-12.0-puddle
    Updated    python-oslo-vmware-lang-2.14.1-1.el7ost.noarch           @rhos-10.0-signed
    Update                             2.23.1-1.el7ost.noarch           @rhelosp-12.0-puddle
    Updated    python-oslotest-2.10.0-1.el7ost.noarch                   @rhos-10.0-signed
    Update                     2.17.1-1.el7ost.noarch                   @rhelosp-12.0-puddle

Comment 5 Alexander Chuzhoy 2018-07-06 13:56:12 UTC

If python-osprofiler is updated to a version from osp12 - the ironic-dbsync doesn't show the error:

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                         Arch                                 Version                                         Repository                                         Size
==============================================================================================================================================================================================
Updating:
 python-osprofiler                               noarch                               1.11.0-1.el7ost                                 rhelosp-12.0-puddle                               115 k

Transaction Summary
==============================================================================================================================================================================================
Upgrade  1 Package

Total download size: 115 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for rhelosp-12.0-puddle
python-osprofiler-1.11.0-1.el7ost.noarch.rpm                                                                                                                           | 115 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : python-osprofiler-1.11.0-1.el7ost.noarch                                                                                                                                   1/2 
  Cleanup    : python-osprofiler-1.4.0-1.el7ost.noarch                                                                                                                                    2/2 
  Verifying  : python-osprofiler-1.11.0-1.el7ost.noarch                                                                                                                                   1/2 
  Verifying  : python-osprofiler-1.4.0-1.el7ost.noarch                                                                                                                                    2/2 

Updated:
  python-osprofiler.noarch 0:1.11.0-1.el7ost                                                                                                                                                  

Complete!
[root@controller-1 ~]# ironic-dbsync --config-file /etc/ironic/ironic.conf upgrade
Option "rpc_backend" from group "DEFAULT" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade dbefd6bdaa2c -> 3d86a077a3f2, add port physical network
INFO  [alembic.runtime.migration] Running upgrade 3d86a077a3f2 -> 868cb606a74a, add version field in base class
[root@controller-1 ~]# echo $?
0

Comment 16 Joanne O'Flynn 2018-08-15 07:56:43 UTC

This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 17 Alexander Chuzhoy 2018-08-21 20:08:22 UTC

Environment:
openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch


The reported issue doesn't reproduce.

Comment 19 errata-xmlrpc 2018-08-29 16:37:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574