Bug 1460711 - Unable to create organization - Katello::Resources::Candlepin::Owner: 400 Bad Request {"displayMessage":"Problem generating ueber cert for owner [NEEDINFO]
Unable to create organization - Katello::Resources::Candlepin::Owner: 400 Bad...
Status: NEW
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Organizations and Locations (Show other bugs)
6.2.9
Unspecified Unspecified
unspecified Severity high (vote)
: Unspecified
: --
Assigned To: satellite6-bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-12 09:25 EDT by Alexander Braverman
Modified: 2017-11-14 10:27 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
bbuckingham: needinfo? (bcourt)


Attachments (Terms of Use)
production.log around error creating custom repo (38.03 KB, text/plain)
2017-06-26 07:56 EDT, Maxim Burgerhout
no flags Details
foreman-tail log from when trying to sync capsule (196.68 KB, text/plain)
2017-08-18 06:13 EDT, Ture Karlsson
no flags Details

  None (edit)
Description Alexander Braverman 2017-06-12 09:25:05 EDT
Description of problem:
As a user with admin permissions, creating a new organization with the name "CS&S" and label "CS_C" failed on the following error:
Error: Task 8c145d72-29e8-4270-8d05-4c79d6f81a3e: RestClient::BadRequest: Katello::Resources::Candlepin::Owner: 400 Bad Request {"displayMessage":"Problem generating ueber cert for owner CS_S","requestUuid":"0bf96ef4-aad1-45bc-a239-843cf875f004"} (POST /candlepin/owners/CS_S/uebercert)

The full trace of the error in foreman/production.log
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in `block in transmit'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/katello/http_resource.rb:98:in `post'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/katello/resources/candlepin.rb:404:in `generate_ueber_cert'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/models/katello/glue/candlepin/owner.rb:46:in `generate_debug_cert'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/models/katello/glue/candlepin/owner.rb:52:in `rescue in load_debug_cert'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/models/katello/glue/candlepin/owner.rb:50:in `load_debug_cert'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/core_ext/object/try.rb:45:in `public_send'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/core_ext/object/try.rb:45:in `try'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/actions/katello/organization/create.rb:31:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/lib/actions/middleware/keep_locale.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:504:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:504:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:419:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:419:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:419:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:419:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:499:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/action.rb:260:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.4/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

After another click, it passed
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
Error

Expected results:
Organization created

Additional info:
Comment 1 Alexander Braverman 2017-06-12 09:27:41 EDT
After creating the organization, edit it, and click on the button next to "Debug certificate": Generate and Download.
It always reproduces the error.
Comment 2 Brad Buckingham 2017-06-13 13:29:04 EDT
Hi Alexander,

Based on the description and comment 1, were you able to create the organization or not?  Was the error during the org create or when doing the 'Generate and Download' of the debug certificate?
Comment 3 Alexander Braverman 2017-06-14 04:39:08 EDT
Hi Brad,

I was able to create the organization after another click on "create".
First time the error appeared when I clicked on "create" during organization creation, second click actually created the organization. Next time I saw this error was when I tried to 'Generate and Download'.
Comment 4 Maxim Burgerhout 2017-06-26 07:56 EDT
Created attachment 1291952 [details]
production.log around error creating custom repo
Comment 5 Maxim Burgerhout 2017-06-26 08:05:48 EDT
Hi Brad,

Have the same error on 6.2.10.

Creating org generates same error as mentioned in comment 1 for me. Need to go into dynflow and skip the "Actions::Katello::Organization::Create" step in order for the task to leave paused/error state.

If you do not skip this task, creating a custom product wasn't possible for me. I have attached attacment 1291952 with the part of my production.log that shows this error.

As this is breaking things that I am sure other people are using too, can we get a prio bump on this?
Comment 6 Ture Karlsson 2017-08-18 06:12:43 EDT
Any progress on this?

I hit the same error on 6.2.11 in my lab, both when creating a new organization and when generating the debug certificate manually. But also when trying to sync a freshly installed Capsule I get "Katello::Resources::Candlepin::Owner: 400 Bad Request {"displayMessage":"Problem generating ueber cert for owner OCP","requestUuid":"919f8efa-99de-4bd0-ae55-7aa38d3b77d1"} (POST /candlepin/owners/OCP/uebercert)"

Attaching foreman-tail from when trying to sync Capsule.
Comment 7 Ture Karlsson 2017-08-18 06:13 EDT
Created attachment 1315157 [details]
foreman-tail log from when trying to sync capsule
Comment 8 Brad Buckingham 2017-09-27 16:49:13 EDT
I have run a test with an internal reference Satellite that is currently running 6.2.12 and was unable to reproduce the error using the CS&S / CS_C example from the original description.

I also did the same on my internal Satellite 6.3 server and was unable to reproduce it.

I do not see any fixes that went in to 6.2.12 that would specifically address the issue; however, it would be interesting to see if the same error occurs after updating.  If so, we may need to get access to the machine to debug to see if it is a case of the Satellite being in a bad state or something else.
Comment 11 Brad Buckingham 2017-10-20 12:58:40 EDT
Barnaby,

Please see info above.  Are you aware any issues with uebercert or fixes that may address this bugzilla?  I was unable to produce it on our internal systems; however, it appears that a few folks have seen issues on various 6.2.z installs.

Eg. error from attachment: 

Katello::Resources::Candlepin::Owner: 400 Bad Request {"displayMessage":"Problem generating ueber cert for owner OCP","requestUuid":"919f8efa-99de-4bd0-ae55-7aa38d3b77d1"} (POST /candlepin/owners/OCP/uebercert) (RestClient::BadRequest)
Comment 12 Barnaby Court 2017-11-14 09:21:42 EST
Brad, There have been so many fixes around uebercert that I could easily see something fixing this issue between 6.2.9 & 6.2.12. However, without having the candlepin log file it's really hard to trace exactly what is going on.
Comment 13 Brad Buckingham 2017-11-14 10:27:27 EST
Barnaby,

It looks like the sosreport in the attached case has a candlepin.log.  Would that help?  If not, we may be able to request an updated sosreport capturing the scenario.  

Also, if you have references to fixes that went in between 6.2.9 and 6.2.12 that would potentially solve the issue, it may help the reporter to work with the customer on upgrading.

Note You need to log in before you can comment on or make changes to this bug.