Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1815540

Summary: ffwd prepare fails with concurrent transaction error
Product: Red Hat OpenStack Reporter: Lukas Bezdicka <lbezdick>
Component: openstack-tripleo-heat-templatesAssignee: David Peacock <dpeacock>
Status: CLOSED ERRATA QA Contact: Lukas Bezdicka <lbezdick>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.0 (Train)CC: dpeacock, jschluet, mburns, morazi, ramishra, sbaker, shardy
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.3.2-0.20200401083149.cd5c992.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-14 12:16:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukas Bezdicka 2020-03-20 14:22:31 UTC
| 1503 | 64c97190-81fc-46e4-9da0-4d92fdd5cc2f | 8dcdfe51-95da-420a-9b52-cb0572d13d90 | SshHostPubKeyDeployment | 2020-03-19 14:28:33 | NULL                | DELETE | FAILED | ConcurrentTransaction_Remote: resources.SshHostPubKeyDeployment: Concurrent transaction for deployments of server 9b6a501b-8f0a-460d-92a8-c9df70941b3e
Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 423, in wrapped
    return func(self, ctx, *args, **kwargs)

  File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 2278, in delete_software_deployment
    cnxt, deployment_id)

  File "/usr/lib/python3.6/site-packages/heat/engine/service_software_config.py", line 389, in delete_software_deployment
    cnxt, sd.server_id, sd.stack_user_project_id)

  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)

  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)

  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 331, in iter
    raise retry_exc.reraise()

  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 167, in reraise
    raise self.last_attempt.result()

  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()

  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception

  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)

  File "/usr/lib/python3.6/site-packages/heat/engine/service_software_config.py", line 127, in _push_metadata_software_deployments
    raise exception.ConcurrentTransaction(action=action)

heat.common.exception.ConcurrentTransaction: Concurrent transaction for deployments of server 9b6a501b-8f0a-460d-92a8-c9df70941b3e
 | 40286a1c-322c-41c0-95fd-23e2953ad410 | {}            | null            | NULL      |          5 | []        | [1504]                                |     NULL |        NULL |                 617 |                      NULL | aaa17b7d-2cd0-4849-b393-d0c1dd5458de |              2163 |         NULL |

Comment 1 David Peacock 2020-03-20 17:12:30 UTC
Working notes:

13 -> 16 PoC upgrade.

MariaDB [heat]> select name,created_at,updated_at,action,status,requires,replaces,replaced_by from resource where id = '1503'\G                                                                                    
*************************** 1. row ***************************
       name: SshHostPubKeyDeployment
 created_at: 2020-03-19 14:28:33
 updated_at: NULL
     action: DELETE
     status: FAILED
   requires: [1504]
   replaces: NULL
replaced_by: NULL
1 row in set (0.001 sec)

MariaDB [heat]> select name,created_at,updated_at,action,status,requires,replaces,replaced_by from resource where id = '1504'\G                                                                                    
*************************** 1. row ***************************
       name: SshHostPubKeyConfig
 created_at: 2020-03-19 14:28:33
 updated_at: NULL
     action: CREATE
     status: COMPLETE
   requires: []
   replaces: NULL
replaced_by: NULL
1 row in set (0.000 sec)

Comment 2 Lukas Bezdicka 2020-03-20 17:58:40 UTC
To reproduce just run on the node as stack:
sh overcloud_upgrade_prepare.sh

Comment 4 David Peacock 2020-03-23 12:48:01 UTC
Hey Lukas,

FYI - Rabi and I are collaborating on this.  Today we poked around the environment you shared and will continue to do so tomorrow, when we regroup (we're in wildly different timezones).

Please keep the environment available to us.

Can you confirm that this is 100% reproducible?

Thanks,
David

Comment 19 errata-xmlrpc 2020-05-14 12:16:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2114