Bug 1469744

Summary: Deadlocks encountered with system.setBaseChannel()
Product: Red Hat Satellite 5 Reporter: Neal Kim <nkim>
Component: ServerAssignee: Grant Gainey <ggainey>
Status: CLOSED ERRATA QA Contact: Martin Korbel <mkorbel>
Severity: high Docs Contact:
Priority: high    
Version: 580CC: dyordano, ggainey, jdobes, mkorbel, ngalvin, nkim, shughes, tkasparek, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: spacewalk-schema-2.5.1-53-sat satellite-schema-5.8.0.36-1-sat spacewalk-backend-2.5.3-154-sat Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1512945 (view as bug list) Environment:
Last Closed: 2017-12-13 07:58:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1450111, 1482563, 1512945    
Attachments:
Description Flags
postgresql log with deadlock transactions none

Description Neal Kim 2017-07-11 18:36:34 UTC
Description of problem:

It seems that in certain cases system.setBaseChannel() will result in deadlocks.

2017-07-11 11:06:47.242 PDT ERROR:  deadlock detected
2017-07-11 11:06:47.242 PDT DETAIL:  Process 12231 waits for ShareLock on transaction 5256920; blocked by process 17706.
	Process 17706 waits for ShareLock on transaction 5256921; blocked by process 12231.
	Process 12231: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17706: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:06:47.242 PDT HINT:  See server log for query details.
2017-07-11 11:06:47.242 PDT CONTEXT:  SQL statement "update rhnPrivateChannelFamily
	           set current_members = rhn_channel.channel_family_current_members(channel_family_id_in, org_id_in),
	               fve_current_members = rhn_channel.cfam_curr_fve_members(channel_family_id_in,org_id_in)
	         where org_id = org_id_in
	           and channel_family_id = channel_family_id_in"
	PL/pgSQL function rhn_channel.update_family_counts(numeric,numeric) line 3 at SQL statement
	SQL statement "SELECT rhn_channel.update_family_counts(channel_family_id_val, server_org_id_val)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 95 at PERFORM

2017-07-11 11:07:19.947 PDT ERROR:  deadlock detected
2017-07-11 11:07:19.947 PDT DETAIL:  Process 17772 waits for ShareLock on transaction 5257528; blocked by process 17774.
	Process 17774 waits for ShareLock on transaction 5257527; blocked by process 17772.
	Process 17772: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17774: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:07:19.947 PDT HINT:  See server log for query details.
2017-07-11 11:07:19.947 PDT CONTEXT:  SQL statement "select pcf.created                     from rhnPrivateChannelFamily pcf
	           inner join rhnserver s on s.org_id = pcf.org_id
	     where s.id = server_id_in
	       for update"
	PL/pgSQL function lock_counts(numeric) line 10 at SQL statement
	SQL statement "SELECT lock_counts(server_id_in)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 60 at PERFORM


Version-Release number of selected component (if applicable):

database-schema-version

5.7.0.27-1.el6sat


How reproducible:

Varies.


Steps to Reproduce:

1. Registered 10k fake clients via ruby-spacewalk (makes the testing easier)
2. Run a script that loops through all the clients and changes the base channel:

client.system.setBaseChannel(key, s.get('id'), 106)
client.system.setBaseChannel(key, s.get('id'), 107)

Probably not a real world example but it seemed to help with triggering the deadlocks.

3. Running two instances of the script staggered seems to improve the likelihood of encountering the deadlocks. 


Actual results:

Deadlocks.


Expected results:

No deadlocks.


Additional info:

Comment 4 Shannon Hughes 2017-07-11 19:45:40 UTC
Created attachment 1296416 [details]
postgresql log with deadlock transactions

deadlock pg log

Comment 23 errata-xmlrpc 2017-12-13 07:58:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3445