Description of problem:

It seems that in certain cases system.setBaseChannel() will result in deadlocks.

2017-07-11 11:06:47.242 PDT ERROR:  deadlock detected
2017-07-11 11:06:47.242 PDT DETAIL:  Process 12231 waits for ShareLock on transaction 5256920; blocked by process 17706.
	Process 17706 waits for ShareLock on transaction 5256921; blocked by process 12231.
	Process 12231: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17706: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:06:47.242 PDT HINT:  See server log for query details.
2017-07-11 11:06:47.242 PDT CONTEXT:  SQL statement "update rhnPrivateChannelFamily
	           set current_members = rhn_channel.channel_family_current_members(channel_family_id_in, org_id_in),
	               fve_current_members = rhn_channel.cfam_curr_fve_members(channel_family_id_in,org_id_in)
	         where org_id = org_id_in
	           and channel_family_id = channel_family_id_in"
	PL/pgSQL function rhn_channel.update_family_counts(numeric,numeric) line 3 at SQL statement
	SQL statement "SELECT rhn_channel.update_family_counts(channel_family_id_val, server_org_id_val)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 95 at PERFORM

2017-07-11 11:07:19.947 PDT ERROR:  deadlock detected
2017-07-11 11:07:19.947 PDT DETAIL:  Process 17772 waits for ShareLock on transaction 5257528; blocked by process 17774.
	Process 17774 waits for ShareLock on transaction 5257527; blocked by process 17772.
	Process 17772: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
	Process 17774: select * from rhn_channel.subscribe_server($1, $2, 0, $3) as result
2017-07-11 11:07:19.947 PDT HINT:  See server log for query details.
2017-07-11 11:07:19.947 PDT CONTEXT:  SQL statement "select pcf.created                     from rhnPrivateChannelFamily pcf
	           inner join rhnserver s on s.org_id = pcf.org_id
	     where s.id = server_id_in
	       for update"
	PL/pgSQL function lock_counts(numeric) line 10 at SQL statement
	SQL statement "SELECT lock_counts(server_id_in)"
	PL/pgSQL function rhn_channel.subscribe_server(numeric,numeric,numeric,numeric,numeric) line 60 at PERFORM


Version-Release number of selected component (if applicable):

database-schema-version

5.7.0.27-1.el6sat


How reproducible:

Varies.


Steps to Reproduce:

1. Registered 10k fake clients via ruby-spacewalk (makes the testing easier)
2. Run a script that loops through all the clients and changes the base channel:

client.system.setBaseChannel(key, s.get('id'), 106)
client.system.setBaseChannel(key, s.get('id'), 107)

Probably not a real world example but it seemed to help with triggering the deadlocks.

3. Running two instances of the script staggered seems to improve the likelihood of encountering the deadlocks. 


Actual results:

Deadlocks.


Expected results:

No deadlocks.


Additional info: