Bug 1655245

OK, this is generic problem and it can be REALLY dangerous.

I just "successfully" run a CV publish _and_ another promote (not to Library) concurrently. Just by firing the 2nd task while the first one was still in Planning phase. It was enough to have 10+ repos in a CV and fire publish + promote (almost) concurrently.


So _any_ type of task suffer from the same. If another task requiring a lock on an object (CV, manifest, host,..) is fired during Planning phase of another task that requires the same locks, _both_ tasks will "acquire" even write lock (or at least tasking system says so to me as an observer).

So one can run more tasks on same object concurrently, with any fatal consequences (delete a CV version while promoting it, update a host concurrently, etc.).

For QE:
much better reproducer (though worth trying both below and original one):

- have "dev" Lifecycle Environment
- publish a CV cv_many_repos with many repos (30 "zoo" pulp repos used in my case)
- once the publish finishes, run concurrently:

hammer content-view publish --name cv_many_repos --organization-id 1 &

hammer content-view version promote --content-view cv_many_repos --version 1.0 --organization-id 1 --to-lifecycle-environment dev &

- check if both tasks will be running concurrently

Another reproducer where I already get corrupted pulp+katello repositories for a CV:

concurrently delete a CV version && promote the same version to an LE.

Both tasks were running concurrently, both failed on an error.

Created redmine issue http://projects.theforeman.org/issues/25608 from this bug

We're checking if a lock is available in active record validation. Since it is not enforced by the database we might run into race condition issues. What might happen is that

1) Task A creates a lock L1 for R1
2) Task B creates a lock L2 for R1
3) Task A checks if L1 is available (it is)
4) Task B checks if L2 is available (it is)
5) Task A acquires L1
6) Task B acquires L2
7) Both tasks hold exclusive lock on R1

A reproducer for usage in the rails console. Just make sure you have at least two tasks and one host.

Expected results:
Only one task holds the locks

Actual results:
Both tasks hold the locks

vvvvv SCRIPT vvvvv

task1 = ForemanTasks::Task.first
task2 = ForemanTasks::Task.last
subject = Host.first

ForemanTasks::Lock.where(:task_id => task1.id).destroy_all
ForemanTasks::Lock.where(:task_id => task2.id).destroy_all
ForemanTasks::Lock.where(:resource_id => subject.id, :resource_type => subject.class.to_s).destroy_all

thread = Thread.new do
  ActiveRecord::Base.transaction do
    ForemanTasks::Lock.exclusive!(subject, task1.id)
    sleep 2
  end
end

ActiveRecord::Base.transaction do
  ForemanTasks::Lock.exclusive!(subject, task2.id)
  sleep 2
end
thread.join

ForemanTasks::Lock.where(:resource_id => subject.id)

^^^^^ SCRIPT ^^^^^

Forgot one thing, to make it easier to reproduce using the script described in #8, edit the active scope in app/models/foreman_tasks/lock.rb to

scope :active, -> { joins(:task) }

Upstream bug assigned to aruzicka

Upstream bug assigned to aruzicka

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/25608 has been resolved.

Verified on Satellite 8.10 sn 10, I didn't reproduce the concurrent lock issue with none of the aforementioned reporoducers

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702