Bug 1496782

Created attachment 1331943 [details]
deploy comman, files and logs

Created attachment 1331944 [details]
sosreport

From the screen log in the tar the deployment seems to fail on mysql going away during deployment.

 Stack overcloud CREATE_FAILED 

overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 6c3b8b30-a414-428d-85cd-d78b4454677f
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    ...
    Debug: Prefetching nova_manage resources for nova_cell_v2
    Debug: Executing: '/usr/bin/nova-manage cell_v2 list_cells --verbose'
    Debug: Storing state
    Debug: Stored state in 0.16 seconds
    Debug: Applying settings catalog for sections reporting, metrics
    Debug: Finishing transaction 120889140
    Debug: Received report to process from controller-0
    Debug: Evicting cache entry for environment 'production'
    Debug: Caching environment 'production' (ttl = 0 sec)
    Debug: Processing report from controller-0 with processor Puppet::Reports::Store
    (truncated, view all with --long)
  deploy_stderr: |
    ...
      File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 484, in _start
        engine_args, maker_args)
      File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 508, in _setup_for_connection
        sql_connection=sql_connection, **engine_kwargs)
      File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 179, in create_engine
        test_conn = _test_connection(engine, max_retries, retry_interval)
      File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 365, in _test_connection
        six.reraise(type(de_ref), de_ref)
      File "<string>", line 2, in reraise
    DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
    (truncated, view all with --long)
Heat Stack create failed.
Heat Stack create failed.

This BZ should be a blocker as the deployment fails and we don't have a workaround.

From customer pov: he can not deploy older version overcloud from current undercloud.

Created attachment 1333761 [details]
new deploy

Created attachment 1333762 [details]
new sos report

Hi Alex.
I saw your comment and realized that I attached wrong files to the BZ.

I add the correct file with 'new' prefix,

Please review them and reassign the BZ as necessary.

Thanks.
Raviv

2017-09-28 07:10:40.814 27653 DEBUG tripleo_common.actions.templates [req-9242beca-9a8a-4f01-84bd-12f69f46cd09 e0d78fa7a4e440b39b433afb033ffea6 309c65c7ac4f4dfc81e919b22f2a6677 - default default] Environments: [{'path': 'overcloud-resource-registry-puppet.yaml'}, {'path': 'environments/docker.yaml'}, {'path': 'environments/docker-ha.yaml'}] run /usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py:323
2017-09-28 07:10:40.967 27653 DEBUG tripleo_common.actions.templates [req-9242beca-9a8a-4f01-84bd-12f69f46cd09 e0d78fa7a4e440b39b433afb033ffea6 309c65c7ac4f4dfc81e919b22f2a6677 - default default] _env_path_is_object https://192.168.24.2:13808/v1/AUTH_309c65c7ac4f4dfc81e919b22f2a6677/overcloud/overcloud-resource-registry-puppet.yaml: True _env_path_is_object /usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py:359
2017-09-28 07:10:46.739 27653 DEBUG tripleo_common.actions.templates [req-9242beca-9a8a-4f01-84bd-12f69f46cd09 e0d78fa7a4e440b39b433afb033ffea6 309c65c7ac4f4dfc81e919b22f2a6677 - default default] _env_path_is_object https://192.168.24.2:13808/v1/AUTH_309c65c7ac4f4dfc81e919b22f2a6677/overcloud/environments/docker.yaml: True _env_path_is_object /usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py:359
2017-09-28 07:10:47.347 27653 DEBUG tripleo_common.actions.templates [req-9242beca-9a8a-4f01-84bd-12f69f46cd09 e0d78fa7a4e440b39b433afb033ffea6 309c65c7ac4f4dfc81e919b22f2a6677 - default default] _env_path_is_object https://192.168.24.2:13808/v1/AUTH_309c65c7ac4f4dfc81e919b22f2a6677/overcloud/environments/docker-ha.yaml: True _env_path_is_object /usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py:359
2017-09-28 07:10:47.363 27653 ERROR tripleo_common.actions.templates [req-9242beca-9a8a-4f01-84bd-12f69f46cd09 e0d78fa7a4e440b39b433afb033ffea6 309c65c7ac4f4dfc81e919b22f2a6677 - default default] Error occurred while processing plan files.: ValueError: The environment is not a valid YAML mapping data type.


I wonder if this is related to the hardcoding of docker deploy for OSP12. Moving over to containers for them to take a look at it.

+'ed the need info for myself and Alex, I think we have answered our "need info" request.

Created attachment 1339873 [details]
Patch to modify tripleo-common to resolve the issue

See attached patch which is under review and should be resolved soon.

Hi Toure,
I have test this bug with on my system and got the same failed result,
The rpm I have is openstack-tripleo-common-7.6.3-0.20171028055750.el7ost.noarch
which is newer and should work,
This bug is also relevent to up scaling the overcloud after undercloud upgrade,
Which is a scenario where customer realize he needs more modes for live migration after he started the upgrade procedure.

New sos report is attached.

Created attachment 1347031 [details]
sosreport

Assigning back to dprince, since he is the one that handled this to begin with.

We're found the issue, in the original patch we looked at the existence of environments/docker.yaml file in the swift plan, however this file *exists* in OSP11. When looking for environments/docker-ha.yaml instead, we get past the error. I'm going to submit a follow-up patch.

openstack-tripleo-common-7.6.3-2.el7ost built including patch for this please check and update status if this resolves it.

Hi,
The patch is not included in osp12 beta puddle,
I have installed the undercloud and have this package:
openstack-tripleo-common-7.6.3-0.20171028055750.el7ost.noarch

But it doe's not include the fix.

Per Jon Schluter's latest comment perhaps the latest build (openstack-tripleo-common-7.6.3-2.el7ost) resolves this?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462