Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1125229

Summary:

:subscriptions :upload - sequential API calls lead to random failures

Product:

Red Hat Satellite

Reporter:

Lukas Pramuk <lpramuk>

Component:

API

Assignee:

Tom McKay <tomckay>

Status:

CLOSED UPSTREAM

QA Contact:

Katello QA List <katello-qa-list>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

Unspecified

CC:

bbuckingham, cwelton

Target Milestone:

Unspecified

Keywords:

Triaged

Target Release:

Unused

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-02-26 14:04:46 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
ruby apipie-bindings reproducer	none
ruby apipie-bindings reproducer	none

Description Lukas Pramuk 2014-07-31 11:26:12 UTC

Created attachment 922878 [details]
ruby apipie-bindings reproducer

Description of problem:
sequential API calls for uploading manifests lead to random failures.
Out of 10 calls only 2 or 3 are successful. The others are failures (likely with older composes) or even have manifests history empty like no calls were placed!

(Yes. The manifests are not the same. 10 unique manifests)

Version-Release number of selected component (if applicable):
Satellite-6.0.4-RHEL-6-20140723.0

How reproducible:
100%

Steps to Reproduce:

0. prepare multiple manifests named manifest-0..9.zip (manifests are unique)

1. use attached apipie-bindings reproducer:
# bundle exec 'reproducer.rb'
Creating organization #0
Creating organization #1
Creating organization #2
Creating organization #3
Creating organization #4
Creating organization #5
Creating organization #6
Creating organization #7
Creating organization #8
Creating organization #9
Uploading manifest #0
Uploading manifest #1
Uploading manifest #2
Uploading manifest #3
Uploading manifest #4
Uploading manifest #5
Uploading manifest #6
Uploading manifest #7
Uploading manifest #8
Uploading manifest #9

Manifest history
[{"statusMessage"=>"org-0 file imported successfully.",
  "status"=>"SUCCESS",
  "created"=>"2014-07-30T18:35:37.370+0000"}]

Manifest history
[{"statusMessage"=>"org-1 file imported successfully.",
  "status"=>"SUCCESS",
  "created"=>"2014-07-30T18:35:42.082+0000"}]

Manifest history
[]

Manifest history
[]

Manifest history
[]

Manifest history
[]

Manifest history
[]

Manifest history
[]

Manifest history
[]

Manifest history
[]


Actual results:
2/10 success

Expected results:
10/10 success

Additional info:
sleep between api calls has magic power to fix things

Comment 1 Lukas Pramuk 2014-07-31 11:38:10 UTC

Actions::Katello::Provider::ManifestImport

Input:

{"provider"=>{"id"=>332, "name"=>"Red Hat"},
 "organization"=>{"id"=>167, "name"=>"org-3", "label"=>"org-3"},
 "path"=>"/usr/share/foreman/tmp/import_9a6f6be2f343aad13bce.zip",
 "force"=>nil,
 "remote_user"=>"admin-305a5f03",
 "locale"=>"en"}

Output:

{}

Exception:

Katello::Errors::CandlepinError: Runtime Error ERROR: deadlock detected
  Detail: Process 9608 waits for ShareLock on transaction 154858; blocked by process 28108.
Process 28108 waits for ExclusiveLock on tuple (5,1) of relation 16582 of database 16386; blocked by process 9608.
  Hint: See server log for query details. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,062

Backtrace:

/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:35:in `rescue in propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:33:in `propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:23:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:25:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:51:in `block (2 levels) in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/util/thread_session.rb:85:in `set_pulp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:41:in `as_pulp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:50:in `block in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/util/thread_session.rb:92:in `set_cp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:36:in `as_cp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:49:in `as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:25:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action/progress.rb:16:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:24:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:35:in `with_locale'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:24:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:442:in `block (2 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:441:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:441:in `block in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `block in with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:436:in `execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:230:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract.rb:152:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract.rb:152:in `with_meta_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
caused by: (RestClient::InternalServerError) Katello::Resources::Candlepin::Owner: 500 Internal Server Error {"displayMessage":"Runtime Error ERROR: deadlock detected\n  Detail: Process 9608 waits for ShareLock on transaction 154858; blocked by process 28108.\nProcess 28108 waits for ExclusiveLock on tuple (5,1) of relation 16582 of database 16386; blocked by process 9608.\n  Hint: See server log for query details. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,062","requestUuid":"9082e8e3-357e-48a0-acf3-8f1a8438fe02"} (POST /candlepin/owners/org-3/imports)
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.28/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.28/lib/restclient_ext/request.rb:44:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/http_resource.rb:111:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/resources/candlepin.rb:388:in `import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:172:in `owner_import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:177:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:127:in `block in process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:114:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:114:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:36:in `on_save'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_9550'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__3542827632456180223__save__3936344273939831370__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:104:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `block in save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:283:in `queue_import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:34:in `import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/katello/provider/manifest_import.rb:42:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:443:in `block (3 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:23:in `block in run'
    ... skipped 61 lines

Comment 2 RHEL Program Management 2014-07-31 11:44:00 UTC

Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 4 Lukas Pramuk 2014-07-31 11:46:40 UTC

Actions::Katello::Provider::ManifestImport

Input:

{"provider"=>{"id"=>334, "name"=>"Red Hat"},
 "organization"=>{"id"=>168, "name"=>"org-4", "label"=>"org-4"},
 "path"=>"/usr/share/foreman/tmp/import_67576e9fdef65a8e88ae.zip",
 "force"=>nil,
 "remote_user"=>"admin-305a5f03",
 "locale"=>"en"}

Output:

{}

Exception:

Katello::Errors::CandlepinError: Runtime Error Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched:81

Backtrace:

/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:35:in `rescue in propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:33:in `propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:23:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:25:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:51:in `block (2 levels) in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/util/thread_session.rb:85:in `set_pulp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:41:in `as_pulp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:50:in `block in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/util/thread_session.rb:92:in `set_cp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:36:in `as_cp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:49:in `as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/remote_action.rb:25:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action/progress.rb:16:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:24:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:35:in `with_locale'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/keep_locale.rb:24:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:442:in `block (2 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:441:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:441:in `block in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `block in with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:365:in `with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:436:in `execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:230:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract.rb:152:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract.rb:152:in `with_meta_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
caused by: (RestClient::InternalServerError) Katello::Resources::Candlepin::Owner: 500 Internal Server Error {"displayMessage":"Runtime Error Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched:81","requestUuid":"251e7130-218d-46a7-9d82-d250a8220080"} (POST /candlepin/owners/org-4/imports)
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.28/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.28/lib/restclient_ext/request.rb:44:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/http_resource.rb:111:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/resources/candlepin.rb:388:in `import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:172:in `owner_import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:177:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:127:in `block in process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:114:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:114:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue.rb:36:in `on_save'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_9550'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__3542827632456180223__save__3936344273939831370__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:104:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `block in save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:283:in `queue_import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/provider.rb:34:in `import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/katello/provider/manifest_import.rb:42:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/action.rb:443:in `block (3 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.2/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:23:in `block in run'
    ... skipped 61 lines

Comment 5 Lukas Pramuk 2015-07-03 11:17:37 UTC

Revisited after 6.1 PB - the bug is still valid.
I've just needed to adapt 'reproducer.rb' to Sat6.1 API

Comment 6 Lukas Pramuk 2015-07-03 11:27:03 UTC

Created attachment 1045828 [details]
ruby apipie-bindings reproducer

Comment 7 Lukas Pramuk 2015-07-03 11:30:09 UTC

Action:

Actions::Katello::Provider::ManifestImport

Input:

{"provider"=>{"id"=>42, "name"=>"Red Hat"},
 "organization"=>{"id"=>22, "name"=>"oorg-6", "label"=>"oorg-6"},
 "path"=>"/usr/share/foreman/tmp/import_747c54c8508579c3e2d6.zip",
 "force"=>nil,
 "remote_user"=>"admin-9a8fdad5",
 "remote_cp_user"=>"admin",
 "locale"=>"en"}

Output:

{}

Exception:

Katello::Errors::CandlepinError: Runtime Error ERROR: deadlock detected
  Detail: Process 43457 waits for ShareLock on transaction 53875; blocked by process 42587.
Process 42587 waits for ExclusiveLock on tuple (0,76) of relation 16581 of database 16387; blocked by process 43457.
  Hint: See server log for query details. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,094

Backtrace:

/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/propagate_candlepin_errors.rb:33:in `rescue in propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/propagate_candlepin_errors.rb:31:in `propagate_candlepin_errors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:27:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:57:in `block (2 levels) in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/katello/util/thread_session.rb:84:in `pulp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:43:in `as_pulp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:56:in `block in as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/katello/util/thread_session.rb:91:in `cp_config'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:38:in `as_cp_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:55:in `as_remote_user'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/remote_action.rb:27:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:16:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/keep_locale.rb:23:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/keep_locale.rb:23:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:480:in `block (2 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `block in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `block in with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:474:in `execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:256:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
caused by: (RestClient::InternalServerError) Katello::Resources::Candlepin::Owner: 500 Internal Server Error {"displayMessage":"Runtime Error ERROR: deadlock detected\n  Detail: Process 43457 waits for ShareLock on transaction 53875; blocked by process 42587.\nProcess 42587 waits for ExclusiveLock on tuple (0,76) of relation 16581 of database 16387; blocked by process 43457.\n  Hint: See server log for query details. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,094","requestUuid":"3c411c2a-4d49-4301-9f06-d59fcd51b2f2"} (POST /candlepin/owners/oorg-6/imports)
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/katello/http_resource.rb:110:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/katello/resources/candlepin.rb:398:in `import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue/provider.rb:153:in `owner_import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue.rb:176:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue.rb:126:in `block in process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue.rb:113:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue.rb:113:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue.rb:35:in `on_save'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_11520'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2038188648126203631__save__1883018000536142230__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:104:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `block in save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue/provider.rb:260:in `queue_import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/models/katello/glue/provider.rb:28:in `import_manifest'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/katello/provider/manifest_import.rb:40:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:481:in `block (3 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.56/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `block in run'
    ... skipped 61 lines