Bug 750307 - Duplicate pools possible
Duplicate pools possible
Status: CLOSED CURRENTRELEASE
Product: Candlepin
Classification: Community
Component: candlepin (Show other bugs)
0.5
Unspecified Solaris
urgent Severity urgent
: ---
: ---
Assigned To: Jesus M. Rodriguez
John Sefler
:
Depends On:
Blocks: rhsm-rhel62
  Show dependency treegraph
 
Reported: 2011-10-31 12:26 EDT by Andrew Edwards
Modified: 2016-04-18 02:00 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-01-13 14:17:08 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andrew Edwards 2011-10-31 12:26:25 EDT
Description of problem:
Candlepin can create duplicate pools if multiple refresh pools request are processed for the same owner at the same time. This issue has been observed in the production environment.

Version-Release number of selected component (if applicable):
0.4.17

How reproducible:
75%. this all depends on getting the refresh pools jobs to execute at the same time.

Steps to Reproduce:
1. Create a new Red Hat user and activate subscriptions. (using Red Hat wapps)

2. issue multiple refresh pools against candlepin *
   http://<candlepin_url_port>/candlepin/owners/<owner_key>/subscriptions?auto_create_owner=true

3. view the pools associated with the newly created owner
   http://<candlepin_url_port>/candlepin/owners/<owner_key>/pools
  
Actual results:
Some pools may be duplicated

Expected results:
the appropriate number of pools are created for the number of subscriptions the owner actually has access to

Additional info:
* While this issue can be duplicated using one instance of candlepin, It is closer to our deployment environment to test with multiple instances of candlepin. 
In this case, we are receiving multiple refresh pools request from an automated process for a single owner. These refresh pool request can be sent to different instances of candlepin and both instances of candlepin will independently process the refresh pools request.
Comment 2 Jesus M. Rodriguez 2011-11-14 17:20:43 EST
The duplication is a result of multiple RefreshPoolsJobs running simultaneously
for the same owner. The fix serializes the jobs for a given owner. Though it is
still valid for multiple jobs to run if they are going to work on different
owners.

single server test
--------------------
1) ensure you have candlepin configured for quartz clustering:

https://fedorahosted.org/candlepin/wiki/QuartzSetup

2) deploy candlepin
3) load up a bunch of subscriptions to allow the refresh pools job to take a bit
of time to run, allowing you to investigate the db to verify things are running
properly

#!/bin/sh
for i in `seq 1 1000`; do curl -k --user admin:admin --request POST --data
'{"product":{"id":"awesomeos-server-basic"},"startDate":"Tue, 13 Sep 2016
01:00:00 -0400","accountNumber":123456,"quantity":20,"endDate":"Wed, 13 Sep 2017
01:00:00 -0400","contractNumber":123,"providedProducts":[{"id":"37060"}]}'
--header 'accept: application/json' --header 'content-type: application/json'
https://localhost:8443/candlepin/owners/admin/subscriptions 1>/dev/null
2>/dev/null; done;


4) open up psql client:
   select id,state,starttime,targetid from cp_job where id like 'refresh%';

5) create a bunch of refresh pools jobs requests (5 is sufficient):

curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions

6) re-run the sql statement from step 3. Notice that only one job is in
   state 2 (RUNNING), the rest are in 0 or 1 (CREATED,PENDING).

7) reload query and monitor the jobs as they go from 0 -> 3, one at at time.

multiple candlepin server test
-------------------------------
1) ensure you have candlepin configured for quartz clustering:

https://fedorahosted.org/candlepin/wiki/QuartzSetup

2) configure both candlepins to talk to the *SAME* database.

jpa.config.hibernate.connection.url=jdbc:postgresql://192.168.1.35/candlepin
org.quartz.dataSource.myDS.URL = jdbc:postgresql://192.168.1.35/candlepin

Do this on BOTH candlepin instances.

3) deploy candlepin on at least 2 machines

4) load up a bunch of subscriptions to allow the refresh pools job to take a bit
of time to run, allowing you to investigate the db to verify things are running
properly

#!/bin/sh
for i in `seq 1 1000`; do curl -k --user admin:admin --request POST --data
'{"product":{"id":"awesomeos-server-basic"},"startDate":"Tue, 13 Sep 2016
01:00:00 -0400","accountNumber":123456,"quantity":20,"endDate":"Wed, 13 Sep 2017
01:00:00 -0400","contractNumber":123,"providedProducts":[{"id":"37060"}]}'
--header 'accept: application/json' --header 'content-type: application/json'
https://localhost:8443/candlepin/owners/admin/subscriptions 1>/dev/null
2>/dev/null; done;

5) open up psql client:
   select id,state,starttime,targetid from cp_job where id like 'refresh%';

6) create a bunch of refresh pools jobs requests (5 is sufficient):

curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions
curl -k -u admin:admin --request PUT
https://localhost:8443/candlepin/owners/admin/subscriptions

7) re-run the sql statement from step 3. Notice that only one job is in
   state 2 (RUNNING), the rest are in 0 or 1 (CREATED,PENDING).

8) reload query and monitor the jobs as they go from 0 -> 3, one at at time.

multiple owners
-----------------
Repeat the setup for loading a bunch of subscriptions for another
owner say, snowwhite.

Then run multiple refresh jobs some for admin and others against
snowwhite.

Verify that only one job from each owner is running.

Note You need to log in before you can comment on or make changes to this bug.