Bug 1746953 - Config-download deployment fails when scaling from 103 to 207 overcloud nodes
Summary: Config-download deployment fails when scaling from 103 to 207 overcloud nodes
Keywords:
Status: CLOSED DUPLICATE of bug 1747126
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Adriano Petrich
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-29 15:09 UTC by Sai Sindhur Malleni
Modified: 2019-08-30 17:17 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-30 17:17:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sai Sindhur Malleni 2019-08-29 15:09:40 UTC
Description of problem:
config-download based deployment fails after heat stack creation and before running ansible when scaling the overcloud from 103 to 202 nodes.

Warning: Permanently added '192.168.3.101' (ECDSA) to the list of known hosts.                                                                                                                            [86/1853]
Warning: Permanently added '192.168.3.93' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.86' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.98' (ECDSA) to the list of known hosts.
ssh admin enablement workflow - FAILED.
Failure caused by error in tasks: create_admin_via_ssh

  create_admin_via_ssh [task_ex_id=4fbe69f4-5361-4b77-9086-e94798f55276] -> Failed to complete action [error=Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB], action=tri
pleo.ansible-playbook, task=write_tmp_playbook]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 37, in on_action_complete
    action.complete(result)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 227, in complete
    self.action_ex.output = self._prepare_output(result).to_dict()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 229, in __set__
    instance_dict(instance), value, None)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 708, in set
    value, old, initiator)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 715, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1893, in wrap
    fn(target, *arg)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 94, in <lambda>
    lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 80, in validate_long_type_length
    raise exc.SizeLimitExceededException(msg)
SizeLimitExceededException: Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB]

    [wf_ex_id=2e4d9cc5-0b17-47ce-87e8-1c2c81f6e134, idx=0]: Failed to complete action [error=Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB], action=tripleo.ansible-pla
ybook, task=write_tmp_playbook]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 37, in on_action_complete
    action.complete(result)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 227, in complete
    self.action_ex.output = self._prepare_output(result).to_dict()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 229, in __set__
    instance_dict(instance), value, None)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 708, in set
    value, old, initiator)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 715, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1893, in wrap
    fn(target, *arg)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 94, in <lambda>
    lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 80, in validate_long_type_length
    raise exc.SizeLimitExceededException(msg)
SizeLimitExceededException: Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB]

Version-Release number of selected component (if applicable):
13

How reproducible:
100%

Steps to Reproduce:
1. Deploy a large overcloud with config-download
2.
3.

Actual results:
Deployment fails after heat stack create and during ssh enablement workflow

Expected results:
Deployment should succeed

Additional info:
2019-08-29 14:37:25Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Deploying overcloud configuration
Enabling ssh admin (tripleo-admin) for hosts:
192.168.3.13 192.168.3.6 192.168.3.11 192.168.3.30 192.168.3.3 192.168.3.17 192.168.3.23 192.168.3.19 192.168.3.5 192.168.3.36 192.168.3.170 192.168.3.153 192.168.3.194 192.168.3.119 192.168.3.159 192.168.3.186 
192.168.3.124 192.168.3.189 192.168.3.135 192.168.3.161 192.168.3.184 192.168.3.107 192.168.3.206 192.168.3.188 192.168.3.214 192.168.3.128 192.168.3.123 192.168.3.104 192.168.3.127 192.168.3.145 192.168.3.148 1
92.168.3.179 192.168.3.208 192.168.3.125 192.168.3.199 192.168.3.204 192.168.3.162 192.168.3.175 192.168.3.88 192.168.3.202 192.168.3.140 192.168.3.134 192.168.3.213 192.168.3.165 192.168.3.120 192.168.3.146 192
.168.3.115 192.168.3.141 192.168.3.160 192.168.3.172 192.168.3.158 192.168.3.137 192.168.3.102 192.168.3.196 192.168.3.177 192.168.3.136 192.168.3.219 192.168.3.200 192.168.3.155 192.168.3.143 192.168.3.118 192.
168.3.173 192.168.3.114 192.168.3.139 192.168.3.117 192.168.3.178 192.168.3.205 192.168.3.192 192.168.3.150 192.168.3.138 192.168.3.105 192.168.3.176 192.168.3.164 192.168.3.207 192.168.3.142 192.168.3.144 192.1
68.3.122 192.168.3.171 192.168.3.109 192.168.3.183 192.168.3.154 192.168.3.187 192.168.3.149 192.168.3.217 192.168.3.157 192.168.3.131 192.168.3.216 192.168.3.108 192.168.3.197 192.168.3.180 192.168.3.190 192.16
8.3.193 192.168.3.211 192.168.3.166 192.168.3.147 192.168.3.132 192.168.3.174 192.168.3.220 192.168.3.169 192.168.3.4 192.168.3.9 192.168.3.8 192.168.3.38 192.168.3.66 192.168.3.61 192.168.3.51 192.168.3.16 192.
168.3.72 192.168.3.37 192.168.3.75 192.168.3.28 192.168.3.20 192.168.3.24 192.168.3.39 192.168.3.82 192.168.3.48 192.168.3.67 192.168.3.77 192.168.3.33 192.168.3.18 192.168.3.58 192.168.3.46 192.168.3.74 192.168
.3.49 192.168.3.29 192.168.3.42 192.168.3.54 192.168.3.21 192.168.3.32 192.168.3.50 192.168.3.31 192.168.3.76 192.168.3.163 192.168.3.111 192.168.3.167 192.168.3.182 192.168.3.130 192.168.3.151 192.168.3.156 192
.168.3.99 192.168.3.191 192.168.3.181 192.168.3.185 192.168.3.198 192.168.3.168 192.168.3.133 192.168.3.152 192.168.3.90 192.168.3.56 192.168.3.15 192.168.3.47 192.168.3.78 192.168.3.34 192.168.3.35 192.168.3.43
 192.168.3.12 192.168.3.44 192.168.3.22 192.168.3.45 192.168.3.25 192.168.3.68 192.168.3.41 192.168.3.14 192.168.3.81 192.168.3.69 192.168.3.57 192.168.3.59 192.168.3.27 192.168.3.52 192.168.3.60 192.168.3.40 19
2.168.3.89 192.168.3.71 192.168.3.55 192.168.3.63 192.168.3.53 192.168.3.10 192.168.3.126 192.168.3.95 192.168.3.110 192.168.3.106 192.168.3.62 192.168.3.87 192.168.3.113 192.168.3.79 192.168.3.73 192.168.3.65 1
92.168.3.70 192.168.3.84 192.168.3.92 192.168.3.121 192.168.3.116 192.168.3.80 192.168.3.83 192.168.3.97 192.168.3.64 192.168.3.100 192.168.3.103 192.168.3.96 192.168.3.94 192.168.3.112 192.168.3.85 192.168.3.91
 192.168.3.101 192.168.3.93 192.168.3.86 192.168.3.98
Using ssh user heat-admin for initial connection.
Using ssh key at /home/stack/.ssh/id_rsa for initial connection.
Inserting TripleO short term key for 192.168.3.13
Inserting TripleO short term key for 192.168.3.6
Inserting TripleO short term key for 192.168.3.11
Inserting TripleO short term key for 192.168.3.30
InsertingWarning: Permanently added '192.168.3.3' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.17' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.23' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.19' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.5' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.36' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.170' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.153' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.194' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.119' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.159' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.186' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.124' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.189' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.135' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.161' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.184' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.107' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.206' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.188' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.214' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.128' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.123' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.104' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.127' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.145' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.148' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.179' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.208' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.125' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.199' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.204' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.162' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.175' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.88' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.202' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.140' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.134' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.213' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.165' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.120' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.146' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.115' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.141' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.160' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.172' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.158' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.137' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.102' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.196' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.177' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.136' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.219' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.200' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.155' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.143' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.118' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.173' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.114' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.139' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.117' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.178' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.205' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.192' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.150' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.138' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.105' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.176' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.164' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.207' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.142' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.144' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.122' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.171' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.109' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.183' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.154' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.187' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.149' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.217' (ECDSA) to the list of known hosts.
 TripleO short term key for 192.168.3.3
Inserting TripleO short term key for 192.168.3.17
Inserting TripleO short term key for 192.168.3.23
Inserting TripleO short term key for 192.168.3.19
Inserting TripleO short term key for 192.168.3.5
Inserting TripleO short term key for 192.168.3.36
Inserting TripleO short term key for 192.168.3.170
Inserting TripleO short term key for 192.168.3.153
Inserting TripleO short term key for 192.168.3.194
Inserting TripleO short term key for 192.168.3.119
Inserting TripleO short term key for 192.168.3.159
Inserting TripleO short term key for 192.168.3.186
Inserting TripleO short term key for 192.168.3.124
Inserting TripleO short term key for 192.168.3.189
Inserting TripleO short term key for 192.168.3.135
Inserting TripleO short term key for 192.168.3.161
Inserting TripleO short term key for 192.168.3.184
Inserting TripleO short term key for 192.168.3.107
Inserting TripleO short term key for 192.168.3.206
Inserting TripleO short term key for 192.168.3.188
Inserting TripleO short term key for 192.168.3.214
Inserting TripleO short term key for 192.168.3.128
Inserting TripleO short term key for 192.168.3.123
Inserting TripleO short term key for 192.168.3.104
Inserting TripleO short term key for 192.168.3.127
Inserting TripleO short term key for 192.168.3.145
Inserting TripleO short term key for 192.168.3.148
Inserting TripleO short term key for 192.168.3.179
Inserting TripleO short term key for 192.168.3.208
Inserting TripleO short term key for 192.168.3.125
Inserting TripleO short term key for 192.168.3.199
Inserting TripleO short term key for 192.168.3.204
Inserting TripleO short term key for 192.168.3.162
Inserting TripleO short term key for 192.168.3.175
Inserting TripleO short term key for 192.168.3.88
Inserting TripleO short term key for 192.168.3.202
Inserting TripleO short term key for 192.168.3.140
Inserting TripleO short term key for 192.168.3.134
Inserting TripleO short term key for 192.168.3.213
Inserting TripleO short term key for 192.168.3.165
Inserting TripleO short term key for 192.168.3.120
Inserting TripleO short term key for 192.168.3.146
Inserting TripleO short term key for 192.168.3.115
Inserting TripleO short term key for 192.168.3.141
Inserting TripleO short term key for 192.168.3.160
Inserting TripleO short term key for 192.168.3.172
Inserting TripleO short term key for 192.168.3.137
Inserting TripleO short term key for 192.168.3.102
Inserting TripleO short term key for 192.168.3.196
Inserting TripleO short term key for 192.168.3.177
Inserting TripleO short term key for 192.168.3.136
Inserting TripleO short term key for 192.168.3.219
Inserting TripleO short term key for 192.168.3.200
Inserting TripleO short term key for 192.168.3.155
Inserting TripleO short term key for 192.168.3.143
Inserting TripleO short term key for 192.168.3.118
Inserting TripleO short term key for 192.168.3.173
Inserting TripleO short term key for 192.168.3.114
Inserting TripleO short term key for 192.168.3.139
Inserting TripleO short term key for 192.168.3.117
Inserting TripleO short term key for 192.168.3.178
Inserting TripleO short term key for 192.168.3.205
Inserting TripleO short term key for 192.168.3.192
Inserting TripleO short term key for 192.168.3.150
Inserting TripleO short term key for 192.168.3.138
Inserting TripleO short term key for 192.168.3.105
Inserting TripleO short term key for 192.168.3.176
Inserting TripleO short term key for 192.168.3.164
Inserting TripleO short term key for 192.168.3.207
Inserting TripleO short term key for 192.168.3.142
Inserting TripleO short term key for 192.168.3.144
Inserting TripleO short term key for 192.168.3.122
Inserting TripleO short term key for 192.168.3.171
Inserting TripleO short term key for 192.168.3.109
Inserting TripleO short term key for 192.168.3.183
Inserting TripleO short term key for 192.168.3.154
Inserting TripleO short term key for 192.168.3.187
Inserting TripleO short term key for 192.168.3.149
Inserting TripleO short term key for 192.168.3.217
Inserting TripleO short term key fWarning: Permanently added '192.168.3.157' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.131' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.216' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.108' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.197' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.180' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.190' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.193' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.211' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.166' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.147' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.132' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.174' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.220' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.169' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.4' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.9' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.8' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.38' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.66' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.61' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.51' (ECDSA) to the list of known hosts.

Warning: Permanently added '192.168.3.72' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.37' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.75' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.28' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.24' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.39' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.82' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.48' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.67' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.77' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.33' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.18' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.58' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.46' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.74' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.49' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.29' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.42' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.54' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.32' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.50' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.31' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.76' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.163' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.111' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.167' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.182' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.130' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.151' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.156' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.99' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.191' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.181' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.185' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.198' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.168' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.133' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.152' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.90' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.56' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.15' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.47' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.78' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.34' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.35' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.43' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.12' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.44' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.45' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.25' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.41' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.14' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.81' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.69' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.57' (ECDSA) to the list of known hosts.
or 192.168.3.157
Inserting TripleO short term key for 192.168.3.131
Inserting TripleO short term key for 192.168.3.216
Inserting TripleO short term key for 192.168.3.108
Inserting TripleO short term key for 192.168.3.197
Inserting TripleO short term key for 192.168.3.180
Inserting TripleO short term key for 192.168.3.190
Inserting TripleO short term key for 192.168.3.193
Inserting TripleO short term key for 192.168.3.211
Inserting TripleO short term key for 192.168.3.166
Inserting TripleO short term key for 192.168.3.147
Inserting TripleO short term key for 192.168.3.132
Inserting TripleO short term key for 192.168.3.174
Inserting TripleO short term key for 192.168.3.220
Inserting TripleO short term key for 192.168.3.169
Inserting TripleO short term key for 192.168.3.4
Inserting TripleO short term key for 192.168.3.9
Inserting TripleO short term key for 192.168.3.8
Inserting TripleO short term key for 192.168.3.38
Inserting TripleO short term key for 192.168.3.66
Inserting TripleO short term key for 192.168.3.61
Inserting TripleO short term key for 192.168.3.51
Inserting TripleO short term key for 192.168.3.16
Inserting TripleO short term key for 192.168.3.72
Inserting TripleO short term key for 192.168.3.37
Inserting TripleO short term key for 192.168.3.75
Inserting TripleO short term key for 192.168.3.28
Inserting TripleO short term key for 192.168.3.20
Inserting TripleO short term key for 192.168.3.24
Inserting TripleO short term key for 192.168.3.39
Inserting TripleO short term key for 192.168.3.82
Inserting TripleO short term key for 192.168.3.48
Inserting TripleO short term key for 192.168.3.67
Inserting TripleO short term key for 192.168.3.77
Inserting TripleO short term key for 192.168.3.33
Inserting TripleO short term key for 192.168.3.18
Inserting TripleO short term key for 192.168.3.58
Inserting TripleO short term key for 192.168.3.46
Inserting TripleO short term key for 192.168.3.74
Inserting TripleO short term key for 192.168.3.49
Inserting TripleO short term key for 192.168.3.29
Inserting TripleO short term key for 192.168.3.42
Inserting TripleO short term key for 192.168.3.54
Inserting TripleO short term key for 192.168.3.21
Inserting TripleO short term key for 192.168.3.32
Inserting TripleO short term key for 192.168.3.50
Inserting TripleO short term key for 192.168.3.31
Inserting TripleO short term key for 192.168.3.76
Inserting TripleO short term key for 192.168.3.163
Inserting TripleO short term key for 192.168.3.111
Inserting TripleO short term key for 192.168.3.167
Inserting TripleO short term key for 192.168.3.182
Inserting TripleO short term key for 192.168.3.151
Inserting TripleO short term key for 192.168.3.156
Inserting TripleO short term key for 192.168.3.99
Inserting TripleO short term key for 192.168.3.191
Inserting TripleO short term key for 192.168.3.181
Inserting TripleO short term key for 192.168.3.185
Inserting TripleO short term key for 192.168.3.198
Inserting TripleO short term key for 192.168.3.168
Inserting TripleO short term key for 192.168.3.133
Inserting TripleO short term key for 192.168.3.152
Inserting TripleO short term key for 192.168.3.90
Inserting TripleO short term key for 192.168.3.56
Inserting TripleO short term key for 192.168.3.15
Inserting TripleO short term key for 192.168.3.47
Inserting TripleO short term key for 192.168.3.78
Inserting TripleO short term key for 192.168.3.34
Inserting TripleO short term key for 192.168.3.35
Inserting TripleO short term key for 192.168.3.43
Inserting TripleO short term key for 192.168.3.12
Inserting TripleO short term key for 192.168.3.44
Inserting TripleO short term key for 192.168.3.22
Inserting TripleO short term key for 192.168.3.45
Inserting TripleO short term key for 192.168.3.25
Inserting TripleO short term key for 192.168.3.68
Inserting TripleO short term key for 192.168.3.41
Inserting TripleO short term key for 192.168.3.14
Inserting TripleO short term key for 192.168.3.81
Inserting TripleO short term key for 192.168.3.69
Inserting TripleO short term key for 192.168.3.57
InseWarning: Permanently added '192.168.3.59' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.27' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.52' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.60' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.40' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.89' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.71' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.55' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.63' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.53' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.10' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.126' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.95' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.110' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.106' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.62' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.87' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.113' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.79' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.73' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.65' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.70' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.84' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.92' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.121' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.116' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.83' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.97' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.64' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.100' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.103' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.94' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.112' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.85' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.91' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.101' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.93' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.86' (ECDSA) to the list of known hosts.
Warning: Permanently added '192.168.3.98' (ECDSA) to the list of known hosts.
ssh admin enablement workflow - FAILED.
Failure caused by error in tasks: create_admin_via_ssh

  create_admin_via_ssh [task_ex_id=4fbe69f4-5361-4b77-9086-e94798f55276] -> Failed to complete action [error=Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB], action=tri
pleo.ansible-playbook, task=write_tmp_playbook]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 37, in on_action_complete
    action.complete(result)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 227, in complete
    self.action_ex.output = self._prepare_output(result).to_dict()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 229, in __set__
    instance_dict(instance), value, None)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 708, in set
    value, old, initiator)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 715, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1893, in wrap
    fn(target, *arg)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 94, in <lambda>
    lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 80, in validate_long_type_length
    raise exc.SizeLimitExceededException(msg)
SizeLimitExceededException: Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB]

    [wf_ex_id=2e4d9cc5-0b17-47ce-87e8-1c2c81f6e134, idx=0]: Failed to complete action [error=Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB], action=tripleo.ansible-pla
ybook, task=write_tmp_playbook]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 37, in on_action_complete
    action.complete(result)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 227, in complete
    self.action_ex.output = self._prepare_output(result).to_dict()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 229, in __set__
    instance_dict(instance), value, None)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 708, in set
    value, old, initiator)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 715, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1893, in wrap
    fn(target, *arg)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 94, in <lambda>
    lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 80, in validate_long_type_length
    raise exc.SizeLimitExceededException(msg)
SizeLimitExceededException: Field size limit exceeded [class=TaskExecution, field=output, size=21645KB, limit=16384KB]


rting TripleO short term key for 192.168.3.59
Inserting TripleO short term key for 192.168.3.27
Inserting TripleO short term key for 192.168.3.52
Inserting TripleO short term key for 192.168.3.60
Inserting TripleO short term key for 192.168.3.40
Inserting TripleO short term key for 192.168.3.89
Inserting TripleO short term key for 192.168.3.71
Inserting TripleO short term key for 192.168.3.55
Inserting TripleO short term key for 192.168.3.63
Inserting TripleO short term key for 192.168.3.53
Inserting TripleO short term key for 192.168.3.10
Inserting TripleO short term key for 192.168.3.126
Inserting TripleO short term key for 192.168.3.95
Inserting TripleO short term key for 192.168.3.106
Inserting TripleO short term key for 192.168.3.62
Inserting TripleO short term key for 192.168.3.87
Inserting TripleO short term key for 192.168.3.113
Inserting TripleO short term key for 192.168.3.79
Inserting TripleO short term key for 192.168.3.73
Inserting TripleO short term key for 192.168.3.65
Inserting TripleO short term key for 192.168.3.70
Inserting TripleO short term key for 192.168.3.84
Inserting TripleO short term key for 192.168.3.92
Inserting TripleO short term key for 192.168.3.121
Inserting TripleO short term key for 192.168.3.116
Inserting TripleO short term key for 192.168.3.80
Inserting TripleO short term key for 192.168.3.83
Inserting TripleO short term key for 192.168.3.97
Inserting TripleO short term key for 192.168.3.64
Inserting TripleO short term key for 192.168.3.100
Inserting TripleO short term key for 192.168.3.103
Inserting TripleO short term key for 192.168.3.96
Inserting TripleO short term key for 192.168.3.94
Inserting TripleO short term key for 192.168.3.112
Inserting TripleO short term key for 192.168.3.85
Inserting TripleO short term key for 192.168.3.91
Inserting TripleO short term key for 192.168.3.101
Inserting TripleO short term key for 192.168.3.93
Inserting TripleO short term key for 192.168.3.86
Inserting TripleO short term key for 192.168.3.98
Starting ssh admin enablement workflow
Started Mistral Workflow tripleo.access.v1.enable_ssh_admin. Execution ID: 8de16d18-dcd6-4ecb-9c2d-8d0c49df9850
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
Removing short term keys locally

Comment 1 Sai Sindhur Malleni 2019-08-29 16:51:19 UTC
Not seeing this error after bumping execution_field_size_limit_kb in /etc/mistral/mistral.conf to 32768 and restarting mistral. However, now SSH enablement workflow is timing out.

Comment 2 James Slagle 2019-08-30 17:17:11 UTC
I think this bug will also be addressed by the fixes for:
https://bugzilla.redhat.com/show_bug.cgi?id=1747126

By raising the limit in mistral.conf, it allowed you to proceed further until you hit the limit on the DB side as shown in the other bug. The fixes for the other bug address the root cause by making the result much smaller.

I'm going to close this as a dupe of bug 1747126.

*** This bug has been marked as a duplicate of bug 1747126 ***


Note You need to log in before you can comment on or make changes to this bug.