Bug 1278612

Summary: specialised queue workers ignore configuration and handle all tasks instead
Product: [Community] Bugzilla Reporter: Sage Grigull <mgrigull>
Component: Bugzilla GeneralAssignee: PnT DevOps Devs <hss-ied-bugs>
Status: CLOSED WORKSFORME QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.4CC: jmcdonal, mtahir, qgong
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-11 04:04:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sage Grigull 2015-11-06 01:08:06 UTC 
Description of problem:
queue-workers appear to be ignoring configuration into dedicated task roles.

Version-Release number of selected component (if applicable):
4.4.10042-7

How reproducible:
assumed 100%

Steps to Reproduce:
1. disable all bugzilla-queue workers
2. enable only a worker that has been configured to handle a dedicated tasks.  e.g. jira
3. observe tasks on all queues decrease

Actual results:
all job types are handled by specialized, dedicated worker

Expected results:
only job types matching configuration are handled by specialized, dedicated workers

Additional info:
Comment 1 Jason McDonald 2015-11-06 01:22:01 UTC 
There was a recent change on production to dispatch emails directly rather than put them through the job queue.  That change was made to clear a big backlog of email jobs, but is still in place.

If that change is going to become permanent, I would suggest reducing the number of job queue processors (to say 4 or 6 per web server) and removing the queue specialization mechanism, which was originally implemented to stop backlogged rules engine jobs from blocking email jobs (which can't happen if the emails aren't going through the job queue).

I estimate that about 90% of the job queue throughput used to be email jobs. Reducing the number of queues by 1/2 or 2/3 would therefore be quite safe, would reduce the load on the database when the Rules Engine kill switch is set, and would reduce the number of pgpool connections permanently consumed by the job queues.
Comment 2 Sage Grigull 2015-11-06 01:26:49 UTC 
Unfortunately I have forgotten the specifics; I do recall that other queue types besides mail were being handled when they shouldn't have.
Comment 3 Jeff Fearn 🐞 2015-11-06 02:09:42 UTC 
IIRC The issue was that non-mail queues where actually processing mail jobs.
Comment 4 Sage Grigull 2015-11-06 03:25:30 UTC 
Given I cannot effectively replicate this issuer at this time, I am happy for this issue to be closed for now.

I can clone it if it pops up again.