Bug 1632106

Summary: neutron db:sync failing during undercloud deploy
Product: Red Hat OpenStack Reporter: Sanjay Upadhyay <supadhya>
Component: openstack-neutronAssignee: Assaf Muller <amuller>
Status: CLOSED CURRENTRELEASE QA Contact: Roee Agiman <ragiman>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: amuller, bcafarel, bhaley, chrisw, jschluet, mburns, nyechiel, slinaber, supadhya
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-14 09:32:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1622577    

Description Sanjay Upadhyay 2018-09-24 06:05:53 UTC 
Description of problem:

2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/bin/neutron-db-manage", line 10, in <module>
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     sys.exit(main())
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return_val |= bool(CONF.command.func(config, CONF.command.name))
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 205, in do_upgrade
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     run_sanity_checks(config, revision)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 670, in run_sanity_checks
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     script_dir.run_env()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     util.load_python_file(self.dir, 'env.py')
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     module = load_module_py(module_id, path)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     mod = imp.load_source(module_id, path, fp)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     run_migrations_online()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     with DBConnection(neutron_config.database.connection, connection) as conn:
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/connection.py", line 32, in __enter__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.engine = session.create_engine(self.connection_url)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     test_conn = _test_connection(engine, max_retries, retry_interval)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return engine.connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self._connection_cls(self, **kwargs)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     if connection is not None else engine.raw_connection()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.pool.unique_connection, _connection)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     e, dialect, self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     util.raise_from_cause(newraise, exc_info)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     reraise(type(exception), exception, tb=exc_tb)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return fn()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return _ConnectionFairy._checkout(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     fairy = _ConnectionRecord.checkout(pool)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     rec = pool._do_get()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self._dec_overflow()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     compat.reraise(exc_type, exc_value, exc_tb)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self._create_connection()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return _ConnectionRecord(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.connection = self.__connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     connection = self.__pool._invoke_creator(self)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return dialect.connect(*cargs, **cparams)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return self.dbapi.connect(*cargs, **cparams)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     return Connection(*args, **kwargs)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self.connect()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     self._request_authentication()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     auth_packet = self._read_packet()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     packet.check_error()
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     err.raise_mysql_exception(self._data)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns:     raise errorclass(errno, errval)
2018-09-24 00:38:41 - Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'iad51.l.root-servers.org' (using password: YES)")
2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]
2018-09-24 00:38:41 - Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade heads returned 1 instead of one of [0]



Version-Release number of selected component (if applicable):
passed_phase1 (2018-09-18.1)


How reproducible:
All the time

Steps to Reproduce:
1. Deploy the undercloud with default options
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Sanjay Upadhyay 2018-09-24 08:30:25 UTC 
Note - 
On a older deployment that this worked and the one I was trying it did not work, the only difference is the undercloud RHEL image - 

Working with - http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/201/images/rhel-guest-image-7.5-201.x86_64.qcow2

Not Working with -http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/7.5/196/images/rhel-guest-image-7.5-196.x86_64.qcow2

we keep testing with latest 7.5 builds, thats how our CI picked up the 201 based rhel image, where the undercloud deployment failed.