1590638 – Packstack Queens all-in-one installation fails with "/usr/bin/openstack domain list --quiet --format csv returned 1: Internal Server Error (HTTP 500)"

RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/

Bug 1590638 - Packstack Queens all-in-one installation fails with "/usr/bin/openstack domain list --quiet --format csv returned 1: Internal Server Error (HTTP 500)"

Summary: Packstack Queens all-in-one installation fails with "/usr/bin/openstack domai...

Keywords:
Status:	NEW
Alias:	None
Product:	RDO
Classification:	Community
Component:	openstack-packstack
Sub Component:
Version:	unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	trunk
Assignee:	Javier Peña
QA Contact:	Shai Revivo
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-06-13 06:01 UTC by Ilja Livenson
Modified:	2018-07-17 13:13 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
10.40.40.3_controller.pp.log (32.00 KB, application/octet-stream) 2018-06-13 06:01 UTC, Ilja Livenson	no flags	Details
openstack-setup.log (6.99 KB, text/plain) 2018-06-13 06:02 UTC, Ilja Livenson	no flags	Details
View All

Description Ilja Livenson 2018-06-13 06:01:00 UTC

Description of problem:

Failing to install packstack --allinone (Queens) on Centos 7.5. Baremetal node, behind an HTTP proxy, http_proxy/no_proxy variables are configured.

It fails with an error:

Copying Puppet modules and manifests                 [ DONE ]
Applying 10.40.40.3_controller.pp
10.40.40.3_controller.pp:                         [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.40.40.3_controller.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Internal Server Error (HTTP 500) (tried 34, for a total of 170 seconds)
You will find full trace in log /var/tmp/packstack/20180612-055313-Ys0MLX/manifests/10.40.40.3_controller.pp.log
Please check log file /var/tmp/packstack/20180612-055313-Ys0MLX/openstack-setup.log for more information


Version-Release number of selected component (if applicable):

Queens

How reproducible:


Steps to Reproduce:
1. Clean OS install
2. Following instructions from https://www.rdoproject.org/install/packstack/
3. Fails after running packstack --allinone

Actual results:

Failed deployment


Expected results:

Successful deployment.

Additional info:

Machine is behind an HTTP proxy, but local resolution and http(s)_proxy/no_proxy variables are configured.

Comment 1 Ilja Livenson 2018-06-13 06:01:46 UTC

Created attachment 1450788 [details]
10.40.40.3_controller.pp.log

Comment 2 Ilja Livenson 2018-06-13 06:02:14 UTC

Created attachment 1450789 [details]
openstack-setup.log

Comment 3 Ilja Livenson 2018-06-13 06:21:20 UTC

Logs from keystone:

2018-06-12 03:38:03.621 43076 CRITICAL keystone [-] Unhandled error: OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'localhost' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
2018-06-12 03:38:03.621 43076 ERROR keystone Traceback (most recent call last):
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/bin/keystone-manage", line 10, in <module>
2018-06-12 03:38:03.621 43076 ERROR keystone     sys.exit(main())
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 44, in main
2018-06-12 03:38:03.621 43076 ERROR keystone     cli.main(argv=sys.argv, config_files=config_files)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1349, in main
2018-06-12 03:38:03.621 43076 ERROR keystone     CONF.command.cmd_class.main()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 830, in main
2018-06-12 03:38:03.621 43076 ERROR keystone     token_manager.flush_expired_tokens()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/token/persistence/backends/sql.py", line 283, in flush_expired_tokens
2018-06-12 03:38:03.621 43076 ERROR keystone     with sql.session_for_write() as session:  # Calls session.begin()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-06-12 03:38:03.621 43076 ERROR keystone     return self.gen.next()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1037, in _transaction_scope
2018-06-12 03:38:03.621 43076 ERROR keystone     context=context) as resource:
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-06-12 03:38:03.621 43076 ERROR keystone     return self.gen.next()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 640, in _session
2018-06-12 03:38:03.621 43076 ERROR keystone     bind=self.connection, mode=self.mode)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 404, in _create_session
2018-06-12 03:38:03.621 43076 ERROR keystone     self._start()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _start
2018-06-12 03:38:03.621 43076 ERROR keystone     engine_args, maker_args)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 515, in _setup_for_connection
2018-06-12 03:38:03.621 43076 ERROR keystone     sql_connection=sql_connection, **engine_kwargs)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2018-06-12 03:38:03.621 43076 ERROR keystone     return wrapped(*args, **kwargs)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine
2018-06-12 03:38:03.621 43076 ERROR keystone     test_conn = _test_connection(engine, max_retries, retry_interval)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection
2018-06-12 03:38:03.621 43076 ERROR keystone     return engine.connect()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2102, in connect
2018-06-12 03:38:03.621 43076 ERROR keystone     return self._connection_cls(self, **kwargs)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
2018-06-12 03:38:03.621 43076 ERROR keystone     if connection is not None else engine.raw_connection()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
2018-06-12 03:38:03.621 43076 ERROR keystone     self.pool.unique_connection, _connection)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-06-12 03:38:03.621 43076 ERROR keystone     e, dialect, self)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-06-12 03:38:03.621 43076 ERROR keystone     util.raise_from_cause(newraise, exc_info)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-06-12 03:38:03.621 43076 ERROR keystone     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-06-12 03:38:03.621 43076 ERROR keystone     return fn()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
2018-06-12 03:38:03.621 43076 ERROR keystone     return _ConnectionFairy._checkout(self)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-06-12 03:38:03.621 43076 ERROR keystone     fairy = _ConnectionRecord.checkout(pool)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2018-06-12 03:38:03.621 43076 ERROR keystone     rec = pool._do_get()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2018-06-12 03:38:03.621 43076 ERROR keystone     self._dec_overflow()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-06-12 03:38:03.621 43076 ERROR keystone     compat.reraise(exc_type, exc_value, exc_tb)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2018-06-12 03:38:03.621 43076 ERROR keystone     return self._create_connection()
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2018-06-12 03:38:03.621 43076 ERROR keystone     return _ConnectionRecord(self)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2018-06-12 03:38:03.621 43076 ERROR keystone     self.__connect(first_connect_check=True)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-06-12 03:38:03.621 43076 ERROR keystone     connection = pool._invoke_creator(self)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-06-12 03:38:03.621 43076 ERROR keystone     return dialect.connect(*cargs, **cparams)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-06-12 03:38:03.621 43076 ERROR keystone     return self.dbapi.connect(*cargs, **cparams)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
2018-06-12 03:38:03.621 43076 ERROR keystone     return Connection(*args, **kwargs)
2018-06-12 03:38:03.621 43076 ERROR keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
2018-06-12 03:38:03.621 43076 ERROR keystone     super(Connection, self).__init__(*args, **kwargs2)
2018-06-12 03:38:03.621 43076 ERROR keystone OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'localhost' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
2018-06-12 03:38:03.621 43076 ERROR keystone 
2018-06-12 03:39:03.684 44047 WARNING oslo_db.sqlalchemy.engines [-] URL mysql://keystone:***@localhost/keystone does not contain a '+drivername' portion, and will make use of a default driver.  A full dbname+drivername:// protocol is recommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility
2018-06-12 03:39:03.692 44047 CRITICAL keystone [-] Unhandled error: OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'localhost' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)

Comment 4 Ilja Livenson 2018-06-13 06:45:36 UTC

Potentially related bug: https://bugs.launchpad.net/packstack/+bug/1768910

Comment 5 Ilja Livenson 2018-07-17 13:13:47 UTC

Update -- adding direct network access, i.e. without http_proxy, solved the issue with deployment.

Note You need to log in before you can comment on or make changes to this bug.