Bug 1475056 - Manifest refresh fails with 'deadlock detected'
Manifest refresh fails with 'deadlock detected'
Status: NEW
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Candlepin (Show other bugs)
6.2.10
x86_64 Linux
high Severity high (vote)
: Unspecified
: --
Assigned To: satellite6-bugs
Katello QA List
sat-prio-proposed
: PrioBumpPM, Triaged
Depends On: 1475886 1481367
Blocks: 1478091
  Show dependency treegraph
 
Reported: 2017-07-25 20:25 EDT by Paul Dudley
Modified: 2018-06-23 01:21 EDT (History)
20 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1475886 1478091 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paul Dudley 2017-07-25 20:25:18 EDT
Description of problem:
Refreshing the manifest on Satellite 6.2.10 results in the following error:

2017-07-21 16:05:26 EDT ERROR:  deadlock detected
2017-07-21 16:05:26 EDT DETAIL:  Process 2848 waits for ShareLock on transaction 231777414; blocked by process 1382.
	Process 1382 waits for ShareLock on transaction 231777394; blocked by process 2848.
	Process 2848: insert into cp_pool_products (created, updated, pool_id, product_id, product_name, dtype, id) values ($1, $2, $3, $4, $5, 'provided', $6)
	Process 1382: insert into cp_entitlement (created, updated, consumer_id, dirty, endDateOverride, owner_id, pool_id, quantity, updatedOnStart, id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)

Version-Release number of selected component (if applicable):
satellite-6.2.10-4.0.el7sat.noarch
candlepin-0.9.54.21-1.el7.noarch
foreman-1.11.0.76-1.el7sat.noarch
katello-3.0.0-20.el7sat.noarch

How reproducible:
So far only on customer Satellite
Comment 11 Paul Dudley 2017-08-04 11:39:17 EDT
Hey Vritant,

That's right - in this case subsequent manifest refreshes continue to fail.
Comment 23 Chris Snyder 2017-08-25 12:46:31 EDT
The following workaround should enable those blocked by this issue to move forward with manifest import.


Approach:

Temporarily disable conflicting, external traffic to the satellite for the duration of manifest import.


Important notes before getting started:
This will result in all attempts to register/unregister or update entitlements from client systems to fail for the duration of manifest import.
After this workaround is completed, operations should return to normal.


Steps to accomplish the above:

1) Add the following to /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf (less the triple quotes):
"""
<Location /rhsm>
  PassengerEnabled off
</Location>
"""

2) Restart httpd: `systemctl restart httpd`

3) Navigate to the manifest import page (Content -> Red Hat Subscriptions -> Click "Manage Manifest" button)

4) Select the file to import using the dialog displayed after clicking "Browse".

5) Click "Upload"

6) Wait awhile (in my case this took somewhere between 30min and 1 hour depending on hardware, it may take longer depending on the size and contents of the manifest)

7) Remove the passage added to /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf in step 1.

8) Restart httpd: `systemctl restart httpd`


After completing the above, the manifest import should succeed.

The above worked for me on my reproducer of the deadlock issue. I believe it will work for others as well.

Note You need to log in before you can comment on or make changes to this bug.