Description of problem: 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/bin/neutron-db-manage", line 10, in <module> 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sys.exit(main()) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return_val |= bool(CONF.command.func(config, CONF.command.name)) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 205, in do_upgrade 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: run_sanity_checks(config, revision) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 670, in run_sanity_checks 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: script_dir.run_env() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: util.load_python_file(self.dir, 'env.py') 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: module = load_module_py(module_id, path) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: mod = imp.load_source(module_id, path, fp) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module> 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: run_migrations_online() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: with DBConnection(neutron_config.database.connection, connection) as conn: 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/neutron/db/migration/connection.py", line 32, in __enter__ 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self.engine = session.create_engine(self.connection_url) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: test_conn = _test_connection(engine, max_retries, retry_interval) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return engine.connect() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return self._connection_cls(self, **kwargs) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__ 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: if connection is not None else engine.raw_connection() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self.pool.unique_connection, _connection) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: e, dialect, self) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: util.raise_from_cause(newraise, exc_info) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: reraise(type(exception), exception, tb=exc_tb) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return fn() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return _ConnectionFairy._checkout(self) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: fairy = _ConnectionRecord.checkout(pool) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: rec = pool._do_get() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self._dec_overflow() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: compat.reraise(exc_type, exc_value, exc_tb) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return self._create_connection() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return _ConnectionRecord(self) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__ 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self.connection = self.__connect() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: connection = self.__pool._invoke_creator(self) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return dialect.connect(*cargs, **cparams) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return self.dbapi.connect(*cargs, **cparams) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: return Connection(*args, **kwargs) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__ 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self.connect() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: self._request_authentication() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: auth_packet = self._read_packet() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: packet.check_error() 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: err.raise_mysql_exception(self._data) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: raise errorclass(errno, errval) 2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'iad51.l.root-servers.org' (using password: YES)") 2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0] 2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0] Version-Release number of selected component (if applicable): passed_phase1 (2018-09-18.1) How reproducible: All the time Steps to Reproduce: 1. Deploy the undercloud with default options 2. 3. Actual results: Expected results: Additional info:
Note - On a older deployment that this worked and the one I was trying it did not work, the only difference is the undercloud RHEL image - Working with - http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/201/images/rhel-guest-image-7.5-201.x86_64.qcow2 Not Working with -http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/196/images/rhel-guest-image-7.5-196.x86_64.qcow2 we keep testing with latest 7.5 builds, thats how our CI picked up the 201 based rhel image, where the undercloud deployment failed.