Bug 1632106 - neutron db:sync failing during undercloud deploy
Summary: neutron db:sync failing during undercloud deploy
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Assaf Muller
QA Contact: Roee Agiman
URL:
Whiteboard:
Depends On:
Blocks: 1622577
TreeView+ depends on / blocked
 
Reported: 2018-09-24 06:05 UTC by Sanjay Upadhyay
Modified: 2019-02-14 09:32 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-14 09:32:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sanjay Upadhyay 2018-09-24 06:05:53 UTC
Description of problem:

2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/bin/neutron-db-manage", line 10, in <module>
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     sys.exit(main())
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return_val |= bool(CONF.command.func(config, CONF.command.name))
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 205, in do_upgrade
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     run_sanity_checks(config, revision)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 670, in run_sanity_checks
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     script_dir.run_env()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     util.load_python_file(self.dir, 'env.py')
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     module = load_module_py(module_id, path)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     mod = imp.load_source(module_id, path, fp)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     run_migrations_online()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     with DBConnection(neutron_config.database.connection, connection) as conn:
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/connection.py", line 32, in __enter__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.engine = session.create_engine(self.connection_url)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     test_conn = _test_connection(engine, max_retries, retry_interval)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return engine.connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self._connection_cls(self, **kwargs)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     if connection is not None else engine.raw_connection()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.pool.unique_connection, _connection)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     e, dialect, self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     util.raise_from_cause(newraise, exc_info)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     reraise(type(exception), exception, tb=exc_tb)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return fn()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return _ConnectionFairy._checkout(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     fairy = _ConnectionRecord.checkout(pool)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     rec = pool._do_get()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self._dec_overflow()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     compat.reraise(exc_type, exc_value, exc_tb)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self._create_connection()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return _ConnectionRecord(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.connection = self.__connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     connection = self.__pool._invoke_creator(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return dialect.connect(*cargs, **cparams)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self.dbapi.connect(*cargs, **cparams)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return Connection(*args, **kwargs)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self._request_authentication()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     auth_packet = self._read_packet()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     packet.check_error()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     err.raise_mysql_exception(self._data)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     raise errorclass(errno, errval)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'iad51.l.root-servers.org' (using password: YES)")
2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]



Version-Release number of selected component (if applicable):
passed_phase1 (2018-09-18.1)


How reproducible:
All the time

Steps to Reproduce:
1. Deploy the undercloud with default options
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Sanjay Upadhyay 2018-09-24 08:30:25 UTC
Note - 
On a older deployment that this worked and the one I was trying it did not work, the only difference is the undercloud RHEL image - 

Working with - http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/201/images/rhel-guest-image-7.5-201.x86_64.qcow2

Not Working with -http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/196/images/rhel-guest-image-7.5-196.x86_64.qcow2

we keep testing with latest 7.5 builds, thats how our CI picked up the 201 based rhel image, where the undercloud deployment failed.


Note You need to log in before you can comment on or make changes to this bug.