Bug 1273257 - Bugzilla job queue does not release pgpool connections
Bugzilla job queue does not release pgpool connections
Status: CLOSED CURRENTRELEASE
Product: Bugzilla
Classification: Community
Component: Internal Tools (Show other bugs)
4.4
Unspecified Unspecified
high Severity high (vote)
: ---
: ---
Assigned To: PnT DevOps Devs
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-20 00:54 EDT by Mark Keir
Modified: 2015-12-15 21:13 EST (History)
2 users (show)

See Also:
Fixed In Version: 4.4.10045.3
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-15 21:13:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mark Keir 2015-10-20 00:54:01 EDT
Description of problem:

The "jobqueue.pl" processes associated with the Bugzilla queue do not release the pgpool frontend connections cleanly on termination.

Version-Release number of selected component (if applicable):


How reproducible:

Have a running "bugzilla-queue" service attaching to a pgpool load balanced postgresql cluster

Steps to Reproduce:
1. jobqueue.pl is running
2. kill -9 {pid of jobqueue.pl}
3. alternate step: service bugzilla-queue stop;service bugzilla-queue start

Actual results:

when process is killed, a connection will be left on the pgpool VIP in a "DISCARD" state which can be show from "ps -efl".  This connection can be reaped by killing the associated process left running on the pgpool frontend server.

alternate step: when cleanly stopped, there are no stray processes left on the pgpool server.  On service restart, a new batch of connections for the bugzilla-queue service are claimed, 12 queues x 2 servers is 24 new connections reaped.

Expected results:

in the alternate step, on clean severance, it would be expected that the connections are released and not result in a step function consumption of the connection pool.

Additional info:
Comment 1 Rony Gong 2015-11-05 02:40:13 EST
Test this on build 4.4.10043-3, Fail:

Firstly, make sure all all job queues are running
1.kill -9 {pid of jobqueue.pl}
2.Check in the pgpool server, find DISCARD

[root@bugzilla-qe-02 /]# ps -efl |grep -i  pool|grep -i bugzilla-qe-06 
 pgpool: bugs bugs bugzilla-qe-06.host.stage.eng.rdu2.redhat.com(39109) DISCARD

3.Then execute: service bugzilla-queue stop;service bugzilla-queue start
in the web server side:

4:Check in the pgool side, find 25 connections from webserver 06, 
The Discard still keep there.
[root@bugzilla-qe-02 /]# ps -efl |grep -i  pool|grep -i bugzilla-qe-06 |wc -l
25

[root@bugzilla-qe-02 /]# ps -efl |grep -i  pool|grep -i bugzilla-qe-06
5 S root     19603 11724  0  80   0 - 62326 poll_s 07:13 ?        00:00:00 pgpool: bugs bugs bugzilla-qe-06.host.stage.eng.rdu2.redhat.com(39109) DISCARD

5.Wait 2 mins, 12 connections lost, butt he Discard still keep there.
[root@bugzilla-qe-02 /]# ps -efl |grep -i  pool|grep -i bugzilla-qe-06 |wc -l
13
Comment 2 Jeff Fearn 2015-11-05 16:51:30 EST
(In reply to Rony Gong from comment #1)
> 4:Check in the pgool side, find 25 connections from webserver 06, 
> The Discard still keep there.

This is not what this patch does, it is impossible for any change in BZ to change the state PGPool is in.

This patch stops the discard state from being created by running `service bugzilla-queue stop`.
Comment 3 Rony Gong 2015-11-08 20:43:11 EST
(In reply to Jeff Fearn from comment #2)
> (In reply to Rony Gong from comment #1)
> > 4:Check in the pgool side, find 25 connections from webserver 06, 
> > The Discard still keep there.
> 
> This is not what this patch does, it is impossible for any change in BZ to
> change the state PGPool is in.
> 
> This patch stops the discard state from being created by running `service
> bugzilla-queue stop`.

As Dev described, this bug fixed now.
Comment 4 Matt Tyson 2015-12-15 21:13:39 EST
This change is now live. If there are any issues, do not reopen this bug. Instead, you should create a new bug and reference this bug.

Note You need to log in before you can comment on or make changes to this bug.