Description of problem: Candlepin can create duplicate pools if multiple refresh pools request are processed for the same owner at the same time. This issue has been observed in the production environment. Version-Release number of selected component (if applicable): 0.4.17 How reproducible: 75%. this all depends on getting the refresh pools jobs to execute at the same time. Steps to Reproduce: 1. Create a new Red Hat user and activate subscriptions. (using Red Hat wapps) 2. issue multiple refresh pools against candlepin * http://<candlepin_url_port>/candlepin/owners/<owner_key>/subscriptions?auto_create_owner=true 3. view the pools associated with the newly created owner http://<candlepin_url_port>/candlepin/owners/<owner_key>/pools Actual results: Some pools may be duplicated Expected results: the appropriate number of pools are created for the number of subscriptions the owner actually has access to Additional info: * While this issue can be duplicated using one instance of candlepin, It is closer to our deployment environment to test with multiple instances of candlepin. In this case, we are receiving multiple refresh pools request from an automated process for a single owner. These refresh pool request can be sent to different instances of candlepin and both instances of candlepin will independently process the refresh pools request.
The duplication is a result of multiple RefreshPoolsJobs running simultaneously for the same owner. The fix serializes the jobs for a given owner. Though it is still valid for multiple jobs to run if they are going to work on different owners. single server test -------------------- 1) ensure you have candlepin configured for quartz clustering: https://fedorahosted.org/candlepin/wiki/QuartzSetup 2) deploy candlepin 3) load up a bunch of subscriptions to allow the refresh pools job to take a bit of time to run, allowing you to investigate the db to verify things are running properly #!/bin/sh for i in `seq 1 1000`; do curl -k --user admin:admin --request POST --data '{"product":{"id":"awesomeos-server-basic"},"startDate":"Tue, 13 Sep 2016 01:00:00 -0400","accountNumber":123456,"quantity":20,"endDate":"Wed, 13 Sep 2017 01:00:00 -0400","contractNumber":123,"providedProducts":[{"id":"37060"}]}' --header 'accept: application/json' --header 'content-type: application/json' https://localhost:8443/candlepin/owners/admin/subscriptions 1>/dev/null 2>/dev/null; done; 4) open up psql client: select id,state,starttime,targetid from cp_job where id like 'refresh%'; 5) create a bunch of refresh pools jobs requests (5 is sufficient): curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions 6) re-run the sql statement from step 3. Notice that only one job is in state 2 (RUNNING), the rest are in 0 or 1 (CREATED,PENDING). 7) reload query and monitor the jobs as they go from 0 -> 3, one at at time. multiple candlepin server test ------------------------------- 1) ensure you have candlepin configured for quartz clustering: https://fedorahosted.org/candlepin/wiki/QuartzSetup 2) configure both candlepins to talk to the *SAME* database. jpa.config.hibernate.connection.url=jdbc:postgresql://192.168.1.35/candlepin org.quartz.dataSource.myDS.URL = jdbc:postgresql://192.168.1.35/candlepin Do this on BOTH candlepin instances. 3) deploy candlepin on at least 2 machines 4) load up a bunch of subscriptions to allow the refresh pools job to take a bit of time to run, allowing you to investigate the db to verify things are running properly #!/bin/sh for i in `seq 1 1000`; do curl -k --user admin:admin --request POST --data '{"product":{"id":"awesomeos-server-basic"},"startDate":"Tue, 13 Sep 2016 01:00:00 -0400","accountNumber":123456,"quantity":20,"endDate":"Wed, 13 Sep 2017 01:00:00 -0400","contractNumber":123,"providedProducts":[{"id":"37060"}]}' --header 'accept: application/json' --header 'content-type: application/json' https://localhost:8443/candlepin/owners/admin/subscriptions 1>/dev/null 2>/dev/null; done; 5) open up psql client: select id,state,starttime,targetid from cp_job where id like 'refresh%'; 6) create a bunch of refresh pools jobs requests (5 is sufficient): curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions 7) re-run the sql statement from step 3. Notice that only one job is in state 2 (RUNNING), the rest are in 0 or 1 (CREATED,PENDING). 8) reload query and monitor the jobs as they go from 0 -> 3, one at at time. multiple owners ----------------- Repeat the setup for loading a bunch of subscriptions for another owner say, snowwhite. Then run multiple refresh jobs some for admin and others against snowwhite. Verify that only one job from each owner is running.