Bug 1384074

Summary: Possible consistency error when auto_execute with large number of tasks
Product: Red Hat Satellite Reporter: Ivan Necas <inecas>
Component: Tasks PluginAssignee: Ivan Necas <inecas>
Status: CLOSED ERRATA QA Contact: Roman Plevka <rplevka>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.0CC: aperotti, bbuckingham, bkearney, ggatward, hyu, inecas, jalviso, jcallaha, jentrena, oshtaier, rplevka, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-dynflow-0.8.13.3-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1394362 (view as bug list) Environment:
Last Closed: 2016-12-12 17:14:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1394362    

Description Ivan Necas 2016-10-12 13:27:34 UTC 
We got some reports around Ruby VM consistency errors, while starting
foreman tasks:

<pre>
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:139: [BUG] Stack consistency error (sp: 303, bp: 302)
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0070 p:0011 s:0303 e:000301 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:139
c:0069 p:0008 s:0299 e:000298 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:655
c:0068 p:0021 s:0296 e:000295 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:878 [FINISH]
c:0067 p:---- s:0292 e:000291 CFUNC  :times
c:0066 p:0010 s:0289 e:000288 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:872
c:0065 p:0014 s:0284 e:000283 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:655
c:0064 p:0037 s:0281 e:000280 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:169
c:0063 p:0048 s:0275 e:000274 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511
c:0062 p:0018 s:0268 e:000267 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335
c:0061 p:0008 s:0266 e:000265 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532
c:0060 p:0007 s:0262 e:000261 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335
c:0059 p:0008 s:0259 e:000258 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250
c:0058 p:0163 s:0256 e:000255 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/connection_pool/threaded.rb:98
c:0057 p:0020 s:0246 e:000245 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250
c:0056 p:0022 s:0242 e:000241 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335
c:0055 p:0059 s:0236 e:000235 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:911
c:0054 p:0030 s:0229 E:0017a8 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:655
c:0053 p:0031 s:0225 E:001bf8 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:139 [FINISH]
c:0052 p:---- s:0221 E:000458 CFUNC  :to_a
c:0051 p:0021 s:0218 e:000217 BLOCK  /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence_adapters/sequel.rb:266
c:0050 p:0013 s:0216 e:000215 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence_adapters/sequel.rb:339
c:0049 p:0016 s:0211 e:000210 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence_adapters/sequel.rb:266
c:0048 p:0011 s:0204 e:000203 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence_adapters/sequel.rb:117
c:0047 p:0011 s:0200 e:000199 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence.rb:80
c:0046 p:0014 s:0195 e:000194 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/execution_plan.rb:409
c:0045 p:0035 s:0187 e:000186 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/execution_plan.rb:341
c:0044 p:0029 s:0180 e:000179 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/persistence.rb:47
c:0043 p:0013 s:0175 e:000174 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/executors/parallel/core.rb:83
c:0042 p:0012 s:0168 e:000166 METHOD /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/executors/parallel/core.rb:18
</pre>

It seems the `to_a` calls are not enough to avoid the dataset's each thread unsafety, and we need to use `all` instead.
Comment 1 Ivan Necas 2016-10-12 13:27:37 UTC 
Created from redmine issue http://projects.theforeman.org/issues/16885
Comment 2 Ivan Necas 2016-10-12 13:27:42 UTC 
Upstream bug assigned to inecas
Comment 4 Bryan Kearney 2016-10-18 10:05:53 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16885 has been resolved.
Comment 20 Roman Plevka 2016-12-06 12:32:01 UTC 
VERIFIED
on
# rpm -qa satellite
satellite-6.2.5-1.0.el7sat.noarch

- no consistency errors present in the dynflow executor logs after completing the test suite.
Comment 21 errata-xmlrpc 2016-12-12 17:14:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940