Bug 1296314

Summary: refresh pools is deleting all pools in master candlepin 2.0.9
Product: [Community] Candlepin Reporter: John Sefler <jsefler>
Component: candlepinAssignee: Michael Stead <mstead>
Status: CLOSED NOTABUG QA Contact: Katello QA List <katello-qa-list>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 1.2CC: jsefler, mstead, vrjain
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-07 23:19:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Sefler 2016-01-06 21:27:03 UTC
Description of problem:
Refreshing pools appears to be deleting all the pools.

I deployed a fresh candlepin standalone instance with TESTDATA=1 from master and immediately after making a PUT call to refresh pools, all the pools disappear. 

I suspect something bad happened in candlepin commit 1665c6dfbe28f8276cbc15a2ff8c8bb340dafdec
    1292877: Do not delete dev pool before we are done with it
    
    Since we were deleting the dev pool before the derived products
    were loaded (lazy), we couldn't look them up when addressing
    modified entitlements.
    
    Moved the dev pool deletion to the end of the remove entitlement
    process to be sure we were done with it.

Version-Release number of selected component (if applicable):
[root@jsefler-f22-candlepin candlepin]# git log | head -1
commit ec0df8e8aefc812c5fd0e57e7f5f2d131ba4371a


How reproducible:


Steps to Reproduce:
1. deploy a standalone candlepin with TESTDATA=1

[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/subscriptions?include=product.id' | python -m json/tool | grep id | wc -l
156

Out of the box there are 156 TESTDATA subscriptions available for your testing pleasure.

[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/pools?include=product.id' | python -m json/tool | grep id | wc -l
124

Out of the box there are 124 TESTDATA pools available for your testing pleasure.

2. refresh the pools (should technically have no effect since I have not changed any subscription/pool)


[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request PUT https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/subscriptions | python -m json/tool
{
    "created": "2016-01-06T21:19:14+0000", 
    "done": false, 
    "finishTime": null, 
    "group": "async group", 
    "id": "refresh_pools_5c58dc22-08e9-429d-999b-2904777e158e", 
    "ownerId": "admin", 
    "principalName": "admin", 
    "result": null, 
    "resultData": null, 
    "startTime": null, 
    "state": "CREATED", 
    "statusPath": "/jobs/refresh_pools_5c58dc22-08e9-429d-999b-2904777e158e", 
    "targetId": "admin", 
    "targetType": "owner", 
    "updated": "2016-01-06T21:19:14+0000"
}

Wait a few seconds for the job state to be FINISHED

[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request GET https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/jobs/refresh_pools_5c58dc22-08e9-429d-999b-2904777e158e | python -m json/tool
{
    "created": "2016-01-06T21:19:14+0000", 
    "done": true, 
    "finishTime": "2016-01-06T21:19:32+0000", 
    "group": "async group", 
    "id": "refresh_pools_5c58dc22-08e9-429d-999b-2904777e158e", 
    "ownerId": "admin", 
    "principalName": "admin", 
    "result": "Pools refreshed for owner Admin Owner", 
    "resultData": "[cloaked]", 
    "startTime": "2016-01-06T21:19:14+0000", 
    "state": "FINISHED", 
    "statusPath": "/jobs/refresh_pools_5c58dc22-08e9-429d-999b-2904777e158e", 
    "targetId": "admin", 
    "targetType": "owner", 
    "updated": "2016-01-06T21:19:32+0000"
}

[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/subscriptions?include=product.id' | python -m json/tool | grep id | wc -l
0

BANG! No more subscriptions

[root@jsefler-6 ~]# curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://jsefler-f22-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/pools?include=product.id' | python -m json/tool | grep id | wc -l
0

BANG! No more pools



Actual results:
  above

Expected results:
  Since I made no changes to the subscriptions/pools, I expected the refresh pools job to effectively be a no-op.  The should not have disappeared.


Additional info:

Comment 1 Michael Stead 2016-01-07 15:02:26 UTC
John, I wasn't able to reproduce this locally. are you absolutely positive that you are running candlepin in standalone mode?

If you are running in hosted mode without the hosted test adapters then this behavior is expected since there is no backing subscription service by default.

Please check to make sure that you are actually running in standalone mode.

Here's the details of my testing.


# Candlepin Config file

$ cat /etc/candlepin/candlepin.conf
# THIS FILE IS AUTOGENERATED.  ANY CHANGES COULD BE OVERWRITTEN.
jpa.config.hibernate.connection.driver_class=org.postgresql.Driver
jpa.config.hibernate.connection.url=jdbc:postgresql:candlepin
jpa.config.hibernate.connection.username=candlepin
jpa.config.hibernate.connection.password=

jpa.config.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
jpa.config.hibernate.show_sql=false

candlepin.auth.trusted.enable=true
candlepin.auth.oauth.enable=true
candlepin.auth.oauth.consumer.rspec.secret=rspec-oauth-secret
candlepin.auth.oauth.consumer.thumbslug.secret=thumbslug-secret

candlepin.importer.fail_on_unknown=false
candlepin.pretty_print=true

candlepin.amqp.enable=false
candlepin.amqp.connect=tcp://localhost:5671?ssl='true'&ssl_cert_alias='candlepin'


# uncomment to enable debug logging in candlepin.log:
#log4j.logger.org.candlepin=DEBUG

candlepin.pinsetter.enable=true

org.quartz.jobStore.misfireThreshold=60000
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.dataSource=myDS
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

org.quartz.dataSource.myDS.driver=org.postgresql.Driver
org.quartz.dataSource.myDS.URL=jdbc:postgresql:candlepin
org.quartz.dataSource.myDS.user=candlepin
org.quartz.dataSource.myDS.password=
org.quartz.dataSource.myDS.maxConnections=5

# Checkng the candlepin status after initial deployment with testdata -- note standalone == true
$ curl -k -u admin:admin https://localhost:8443/candlepin/status
{
  "result" : true,
  "version" : "2.0.10",
  "rulesVersion" : "5.19",
  "release" : "1",
  "standalone" : true,
  "timeUTC" : "2016-01-07T14:44:15+0000",
  "managerCapabilities" : [ "cores", "ram", "instance_multiplier", "derived_product", "cert_v3", "guest_limit", "vcpu", "hypervisors_async", "storage_band", "remove_by_pool_id" ],
  "rulesSource" : "DEFAULT"
}


$ curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://localhost:8443/candlepin/owners/admin/subscriptions?include=product.id' | python -m json/tool | grep id | wc -l
156

curl -stderr /dev/null --insecure --user admin:admin --request GET 'https://localhost:8443/candlepin/owners/admin/pools?include=product.id' | python -m json/tool | grep id | wc -l
124

# Initial counts look ok -- run refresh pools.
$ curl --insecure --user admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions
$

No job in the response because refresh was ignored by candlepin. Looking at the candlepin logs we see:

2016-01-07 10:46:45,231 [thread=http-bio-8443-exec-8] [req=3e8f8f0e-1f1a-465d-8d12-a71aa15a819e, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=PUT, uri=/candlepin/owners/admin/subscriptions
2016-01-07 10:46:45,239 [thread=http-bio-8443-exec-8] [req=3e8f8f0e-1f1a-465d-8d12-a71aa15a819e, org=] WARN  org.candlepin.resource.OwnerResource - Ignoring refresh pools request due to standalone config.


# Checking the counts again, they are all still there.
$ curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://localhost:8443/candlepin/owners/admin/subscriptions?include=product.id' | python -m json/tool | grep id | wc -l
156

curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://localhost:8443/candlepin/owners/admin/pools?include=product.id' | python -m json/tool | grep id | wc -l
124

Comment 2 John Sefler 2016-01-07 23:19:36 UTC
Thank you mstead for your assistance.

The issue that is outlined in comment 0 occurs when candlepin.conf is manually pre-configured with candlepin.standalone=false and re-deployed without any adapters.  The correct way to deploy an onpremise candlepin server with candlepin.standalone=false is to also include configuration module.config.hosted.configuration.module=org.candlepin.hostedtest.AdapterOverrideModule and include the -h option to the deploy script.  This is applicable to candlepin-2.0.10-1 and above.   Deploying with HOSTEDTEST="hostedtest" should effectively run candlepin with a fake adapter that will prevent a call to refresh pools from deleting all the TESTDATA subscription/pools.

Moving to CLOSED NOTABUG

Comment 3 John Sefler 2016-01-08 20:53:13 UTC
Here the definitive doc that explains how to configure/deploy candlepin in a hosted mode for testing purposes...

http://www.candlepinproject.org/docs/candlepin/mode_agnostic_spec_testing.html