Bug 1469744 - Deadlocks encountered with system.setBaseChannel()
Deadlocks encountered with system.setBaseChannel()
Status: NEW
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server (Show other bugs)
570
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Grant Gainey
Red Hat Satellite QA List
:
Depends On:
Blocks: sat58-errata
  Show dependency treegraph
 
Reported: 2017-07-11 14:36 EDT by Neal Kim
Modified: 2017-08-11 17:10 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
postgresql log with deadlock transactions (12.12 KB, text/plain)
2017-07-11 15:45 EDT, Shannon Hughes
no flags Details

  None (edit)
Description Neal Kim 2017-07-11 14:36:34 EDT
Description of problem:

It seems that in certain cases system.setBaseChannel() will result in deadlocks.

2017-07-11 11:06:47.242 PDT ERROR:  deadlock detected
2017-07-11 11:06:47.242 PDT DETAIL:  Process 12231 waits for ShareLock on transaction 5256920; blocked by process 17706.
	Process 17706 waits for ShareLock on transaction 5256921; blocked by process 12231.
	Process 12231: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17706: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:06:47.242 PDT HINT:  See server log for query details.
2017-07-11 11:06:47.242 PDT CONTEXT:  SQL statement "update rhnPrivateChannelFamily
	           set current_members = rhn_channel.channel_family_current_members(channel_family_id_in, org_id_in),
	               fve_current_members = rhn_channel.cfam_curr_fve_members(channel_family_id_in,org_id_in)
	         where org_id = org_id_in
	           and channel_family_id = channel_family_id_in"
	PL/pgSQL function rhn_channel.update_family_counts(numeric,numeric) line 3 at SQL statement
	SQL statement "SELECT rhn_channel.update_family_counts(channel_family_id_val, server_org_id_val)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 95 at PERFORM

2017-07-11 11:07:19.947 PDT ERROR:  deadlock detected
2017-07-11 11:07:19.947 PDT DETAIL:  Process 17772 waits for ShareLock on transaction 5257528; blocked by process 17774.
	Process 17774 waits for ShareLock on transaction 5257527; blocked by process 17772.
	Process 17772: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17774: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:07:19.947 PDT HINT:  See server log for query details.
2017-07-11 11:07:19.947 PDT CONTEXT:  SQL statement "select pcf.created                     from rhnPrivateChannelFamily pcf
	           inner join rhnserver s on s.org_id = pcf.org_id
	     where s.id = server_id_in
	       for update"
	PL/pgSQL function lock_counts(numeric) line 10 at SQL statement
	SQL statement "SELECT lock_counts(server_id_in)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 60 at PERFORM


Version-Release number of selected component (if applicable):

database-schema-version

5.7.0.27-1.el6sat


How reproducible:

Varies.


Steps to Reproduce:

1. Registered 10k fake clients via ruby-spacewalk (makes the testing easier)
2. Run a script that loops through all the clients and changes the base channel:

client.system.setBaseChannel(key, s.get('id'), 106)
client.system.setBaseChannel(key, s.get('id'), 107)

Probably not a real world example but it seemed to help with triggering the deadlocks.

3. Running two instances of the script staggered seems to improve the likelihood of encountering the deadlocks. 


Actual results:

Deadlocks.


Expected results:

No deadlocks.


Additional info:
Comment 4 Shannon Hughes 2017-07-11 15:45 EDT
Created attachment 1296416 [details]
postgresql log with deadlock transactions

deadlock pg log

Note You need to log in before you can comment on or make changes to this bug.