Bug 726784

Summary: sharding scenario is failing to migrate an owner from one candlepin to another
Product: [Community] Candlepin Reporter: John Sefler <jsefler>
Component: candlepinAssignee: Jesus M. Rodriguez <jesusr>
Status: CLOSED CURRENTRELEASE QA Contact: John Sefler <jsefler>
Severity: high Docs Contact:
Priority: unspecified    
Version: 0.5CC: bkearney, dgoodwin, jesusr, spandey
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-17 13:04:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 682238    

Description John Sefler 2011-07-29 19:29:16 UTC 
Description of problem:
This is a possible duplicate of bug 691870 which we are having difficulty verifying.

Here's my test scenario...

I've got two on premises candlepin servers depolyed with TESTDATA=1

On server jsefler-onprem-62candlepin I will delete owner donaldduck and will then attempt to migrate existing owner donalduck from mgmt5 to jsefler-onprem-62candlepin.

Here we go....

[jsefler@jsefler ~]$ curl -k -u admin:admin -X DELETE "https://jsefler-onprem-62candlepin.usersys.redhat.com:8443/candlepin/owners/donaldduck"
[jsefler@jsefler ~]$ curl -k -u admin:admin "https://jsefler-onprem-62candlepin.usersys.redhat.com:8443/candlepin/owners" | python -mjson.tool  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
101   506  101   506    0     0   2122      0 --:--:-- --:--:-- --:--:--  4600
[
    {
        "contentPrefix": null, 
        "created": "2011-07-29T18:12:46.324+0000", 
        "displayName": "Admin Owner", 
        "href": "/owners/admin", 
        "id": "8a90f8c6317718cc0131771936750006", 
        "key": "admin", 
        "parentOwner": null, 
        "updated": "2011-07-29T18:12:46.324+0000", 
        "upstreamUuid": null
    }, 
    {
        "contentPrefix": null, 
        "created": "2011-07-29T18:12:47.132+0000", 
        "displayName": "Snow White", 
        "href": "/owners/snowwhite", 
        "id": "8a90f8c6317718cc01317719399c000a", 
        "key": "snowwhite", 
        "parentOwner": null, 
        "updated": "2011-07-29T18:12:47.132+0000", 
        "upstreamUuid": null
    }
]
[jsefler@jsefler ~]$

^^^ donalduck is now deleted from jsefler-onprem-62candlepin


[jsefler@jsefler ~]$ curl -k -u admin:admin "https://mgmt5.rhq.lab.eng.bos.redhat.com:8443/candlepin/owners/donaldduck" | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
129   258  129   258    0     0   1433      0 --:--:-- --:--:-- --:--:--  4372
{
    "contentPrefix": null, 
    "created": "2011-07-29T18:36:41.591+0000", 
    "displayName": "Donald Duck", 
    "href": "/owners/donaldduck", 
    "id": "8a90f88f31772f010131772f1cf7000e", 
    "key": "donaldduck", 
    "parentOwner": null, 
    "updated": "2011-07-29T18:36:41.591+0000", 
    "upstreamUuid": null
}
[jsefler@jsefler ~]$ 

^^^ donaldduck exists on  mgmt5, now lets try to migrate owner donaldduck from mgmt5 to jsefler-onprem-62candlepin...

[jsefler@jsefler ~]$ curl -u admin:admin -k -X POST "https://jsefler-onprem-62candlepin.usersys.redhat.com:8443/candlepin/migrations?entity=owner&id=8a90f88f31772f010131772f1cf7000e&delete=false&uri=mgmt5.rhq.lab.eng.bos.redhat.com:8443" | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
131   393  131   393    0     0    814      0 --:--:-- --:--:-- --:--:--  2673
{
    "created": "2011-07-29T19:01:41.921+0000", 
    "finishTime": null, 
    "group": "Pinsetter Long Running Job Group", 
    "id": "migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa", 
    "ownerKey": "8a90f88f31772f010131772f1cf7000e", 
    "principalName": "admin", 
    "result": null, 
    "startTime": null, 
    "state": "CREATED", 
    "statusPath": "/jobs/migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa", 
    "updated": "2011-07-29T19:01:41.921+0000"
}
[jsefler@jsefler ~]$ curl -u admin:admin -k "https://jsefler-onprem-62candlepin.usersys.redhat.com:8443/candlepin/jobs/migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa" | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
111   670  111   670    0     0   4041      0 --:--:-- --:--:-- --:--:-- 17631
{
    "created": "2011-07-29T19:01:41.921+0000", 
    "finishTime": null, 
    "group": "Pinsetter Long Running Job Group", 
    "id": "migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa", 
    "ownerKey": "8a90f88f31772f010131772f1cf7000e", 
    "principalName": "admin", 
    "result": "org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response]", 
    "startTime": "2011-07-29T19:01:41.928+0000", 
    "state": "FAILED", 
    "statusPath": "/jobs/migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa", 
    "updated": "2011-07-29T19:01:42.202+0000"
}
[jsefler@jsefler ~]$ 

^^^^ Note the job result shows an exception coming from mgmt5



Here's the exception in catalina.out on jsefler-onprem-62candlepin...
Note: I do not see any exception/nor contact from jsefler-onprem-62candlepin in catalina.out on mgmt5


Jul 29 15:01:42 [QuartzScheduler_Worker-1] INFO  org.fedoraproject.candlepin.pinsetter.tasks.MigrateOwnerJob - Migrating owner [8a90f88f31772f010131772f1cf7000e] from candlepin instance running on [http://mgmt5.rhq.lab.eng.bos.redhat.com:8443/candlepin]
Jul 29 15:01:42 [QuartzScheduler_Worker-1] ERROR org.quartz.core.JobRunShell - Job Pinsetter Long Running Job Group.migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa threw an unhandled Exception: 
java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response
	at org.jboss.resteasy.client.core.ClientInvoker.invoke(ClientInvoker.java:96)
	at org.jboss.resteasy.client.core.ClientProxy.invoke(ClientProxy.java:59)
	at $Proxy142.replicateOwner(Unknown Source)
	at org.fedoraproject.candlepin.pinsetter.tasks.MigrateOwnerJob.replicateOwner(MigrateOwnerJob.java:238)
	at org.fedoraproject.candlepin.pinsetter.tasks.MigrateOwnerJob.execute(MigrateOwnerJob.java:141)
	at org.fedoraproject.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:59)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
Caused by: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1987)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.jboss.resteasy.client.core.executors.ApacheHttpClientExecutor.execute(ApacheHttpClientExecutor.java:75)
	at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:39)
	at org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPInterceptor.execute(AcceptEncodingGZIPInterceptor.java:40)
	at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:45)
	at org.jboss.resteasy.client.ClientRequest.execute(ClientRequest.java:449)
	at org.jboss.resteasy.client.ClientRequest.httpMethod(ClientRequest.java:679)
	at org.jboss.resteasy.client.core.ClientInvoker.invoke(ClientInvoker.java:92)
	... 7 more
Jul 29 15:01:42 [QuartzScheduler_Worker-1] ERROR org.quartz.core.ErrorLogger - Job (Pinsetter Long Running Job Group.migrate_owner_9dc9eabd-0c66-46d1-ba7e-9ce421b484aa threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:210)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
Caused by: java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response
	at org.jboss.resteasy.client.core.ClientInvoker.invoke(ClientInvoker.java:96)
	at org.jboss.resteasy.client.core.ClientProxy.invoke(ClientProxy.java:59)
	at $Proxy142.replicateOwner(Unknown Source)
	at org.fedoraproject.candlepin.pinsetter.tasks.MigrateOwnerJob.replicateOwner(MigrateOwnerJob.java:238)
	at org.fedoraproject.candlepin.pinsetter.tasks.MigrateOwnerJob.execute(MigrateOwnerJob.java:141)
	at org.fedoraproject.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:59)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
	... 1 more
Caused by: org.apache.commons.httpclient.ProtocolException: The server mgmt5.rhq.lab.eng.bos.redhat.com failed to respond with a valid HTTP response
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1987)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.jboss.resteasy.client.core.executors.ApacheHttpClientExecutor.execute(ApacheHttpClientExecutor.java:75)
	at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:39)
	at org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPInterceptor.execute(AcceptEncodingGZIPInterceptor.java:40)
	at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:45)
	at org.jboss.resteasy.client.ClientRequest.execute(ClientRequest.java:449)
	at org.jboss.resteasy.client.ClientRequest.httpMethod(ClientRequest.java:679)
	at org.jboss.resteasy.client.core.ClientInvoker.invoke(ClientInvoker.java:92)
	... 7 more






Version-Release number of selected component (if applicable):
[root@jsefler-onprem-62candlepin proxy]# git show-ref | grep master
04279aa2a44d356211a34069511f02d1e82a1a49 refs/heads/master
04279aa2a44d356211a34069511f02d1e82a1a49 refs/remotes/origin/master
Comment 3 Bryan Kearney 2012-07-17 13:04:25 UTC 
Marking all community bugs modified or beyong as closed.