Bug 1192581 - hammer-cli-import broken due to API changes in Sat6.1
Summary: hammer-cli-import broken due to API changes in Sat6.1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Transitions
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: Unspecified
Assignee: Grant Gainey
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-13 18:37 UTC by Grant Gainey
Modified: 2017-02-23 20:30 UTC (History)
8 users (show)

Fixed In Version: rubygem-hammer_cli_import-0.10.9.1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 05:25:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
hammer_cli_import-0.10.6.gem (74.50 KB, application/octet-stream)
2015-02-16 17:51 UTC, Grant Gainey
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1592 0 normal SHIPPED_LIVE Important: Red Hat Satellite 6.1.1 on RHEL 6 2015-08-12 09:04:35 UTC

Description Grant Gainey 2015-02-13 18:37:34 UTC
Description of problem:

Once BZ#1192556 is fixed and hammer-cli-import can talk to the Sat6 server, it breaks due to API changes.

Version-Release number of selected component (if applicable):

hammer-cli-import 0.10.6
Sat6-nightly (13-FEB-2015)

How reproducible:

- Update hammer-cli-import to 0.10.6
- Attempt to import exported organizations:
- Get following traceback and be sad:

[root@ibm-ls21-04 hammer-cli-import]# hammer -v import organization --csv-file /tmp/exports/users.csv --debug --verbose
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:68: warning: already initialized constant DEFAULT_LABEL_INDENT
Importing from /tmp/exports/users.csv
Creating new organization: RED HAT SATELLITE ENGINEERING
entity_hash: {:description=>"Imported 'RED HAT SATELLITE ENGINEERING' organization from Red Hat Satellite 5", :name=>"RED HAT SATELLITE ENGINEERING", :id=>1}
Caught ApipieBindings::MissingArgumentsError:ApipieBindings::MissingArgumentsError: name while processing CSV line:
{"organization_id"=>"1", "organization"=>"RED HAT SATELLITE ENGINEERING"}
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/action.rb:62:in `validate!'
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/api.rb:158:in `call'
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/resource.rb:14:in `call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:127:in `api_call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:139:in `api_call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:144:in `mapped_api_call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:425:in `_create_entity'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:379:in `create_entity'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/organization.rb:84:in `import_single_row'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:520:in `call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:520:in `cvs_iterate'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/importtools.rb:302:in `call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/importtools.rb:302:in `handle_missing_and_supress'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:519:in `cvs_iterate'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/csvhelper.rb:52:in `csv_each'
/usr/lib/ruby/gems/1.8/gems/fastercsv-1.5.4/lib/faster_csv.rb:1545:in `each'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/csvhelper.rb:43:in `csv_each'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:518:in `cvs_iterate'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:526:in `import'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:562:in `execute'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:23:in `run'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:23:in `run'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:23:in `run'
/usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/bin/hammer:108
/usr/bin/hammer:19:in `load'
/usr/bin/hammer:19
Summary
  No action taken.

Actual results:

Traceback

Expected results:

Entities created

Additional info:

This all Just Works in 6.0.x. Something in 6.1 has invalidated hammer-cli-import.

Comment 2 Grant Gainey 2015-02-13 19:50:22 UTC
Output from /var/log/foreman/production.log while getting the above traceback:
===
Processing by Katello::Api::V2::OrganizationsController#index as application/json;version=2
  Parameters: {"per_page"=>"999999", "api_version"=>"v2", "organization"=>{}}
Authorized user admin(Admin User)
  Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/views/katello/api/v2/organizations/index.json.rabl within api/v2/layouts/index_layout (6.5ms)
Completed 200 OK in 53ms (Views: 7.9ms | ActiveRecord: 13.9ms)
===

The code to call the API is this:
===
      # Call API. Ideally accessed via +api_call+ instance method.
      # This is supposed to be the only way to access @api.
      def api_call(resource, action, params = {}, headers = {}, dbg = false)
        @api.resource(resource).call(action, params, headers)
      rescue
        puts("Error on api.resource(#{resource.inspect}).call(#{action.inspect}, #{params.inspect}):")
        raise
      end

===

which is failing like this:
===
Error on api.resource(:organizations).call(:create, {:description=>"Imported 'RED HAT SATELLITE ENGINEERING' organization from Red Hat Satellite 5", :name=>"RED HAT SATELLITE ENGINEERING", :id=>1}):
===

because of this error-return:
===
Caught ApipieBindings::MissingArgumentsError:ApipieBindings::MissingArgumentsError: name
===

Comment 3 Grant Gainey 2015-02-13 20:44:30 UTC
As a test, modified the call-code for organization-create specifically to be this:
===
        p = {'organization'=>params}
        @api.resource(resource).call(action, p, headers)
===

and it failed with the same error:
===
Caught ApipieBindings::MissingArgumentsError:ApipieBindings::MissingArgumentsError: name
===

while spitting out the following debug output:
===
Error on api.resource(:organizations).call(:create, {"organization"=>{:description=>"Imported 'RED HAT SATELLITE ENGINEERING' organization from Red Hat Satellite 5", :name=>"RED HAT SATELLITE ENGINEERING", :id=>1}}):
===

and the following production.log statement:
===
Processing by Katello::Api::V2::OrganizationsController#index as application/json;version=2
  Parameters: {"organization"=>{}, "api_version"=>"v2"}
Authorized user admin(Admin User)
  Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0/app/views/katello/api/v2/organizations/index.json.rabl within api/v2/layouts/index_layout (7.1ms)
Completed 200 OK in 50ms (Views: 9.2ms | ActiveRecord: 10.5ms)
===

Comment 4 Martin Bacovsky 2015-02-16 10:03:30 UTC
The cause of this error is a validation of parameters that was added to apipie-bindings recently. It currently checks just required parameters according to apipie docs.

In this case the problem originates in somewhat inconsistent API parameter format in Katello and Foreman. Unfortunately Organization is resource from Foreman that is extended in Katello and it has both nested and not nested parameter format (See https://<your-satellite>/apidoc/v2/organizations/create.html) I believe that Organization PUT/POST is the only resource where the problem is present.

The solution for the issue you are experiencing would be to fix the parameters you are sending to match the API documentation (or fix the API/API documentation). Minimal Org params should look like:

<pre>
{
            "name" => "myOrg",
    "organization" => {
        "name" => "myOrg"
    }
}
</pre>

Comment 5 Grant Gainey 2015-02-16 17:51:33 UTC
Created attachment 992298 [details]
hammer_cli_import-0.10.6.gem

jsherrill requested a reproducer, so here's one.

Install the attached gem, hammer_cli_import-0.10.6.gem, in order to get past the problem in BZ#1192581

Put the below into a file, /tmp/users.csv:

===
organization_id,organization,user_id,username,last_name,first_name,position,email,role,creation_time,last_login_time,active
1,RED HAT SATELLITE ENGINEERING,1,admin,Nimda,Admin,,root@localhost,Organization Administrator;Satellite Administrator,2015-01-28 18:38:36,2015-02-11 19:13:25,enabled
===

and then run the following import command:

===
hammer import organization --csv-file /tmp/users.csv --debug --verbose
===

and watch it fail:

===
(1192581) ~/hammer-cli-import $ hammer import organization --csv-file /tmp/exports/users.csv --debug --verbose 
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:68: warning: already initialized constant DEFAULT_LABEL_INDENT
Importing from /tmp/exports/users.csv
Creating new organization: RED HAT SATELLITE ENGINEERING
entity_hash: {:description=>"Imported 'RED HAT SATELLITE ENGINEERING' organization from Red Hat Satellite 5", :name=>"RED HAT SATELLITE ENGINEERING", :id=>1}
Caught ApipieBindings::MissingArgumentsError:ApipieBindings::MissingArgumentsError: name while processing CSV line:
{"organization_id"=>"1", "organization"=>"RED HAT SATELLITE ENGINEERING"}
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/action.rb:62:in `validate!'
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/api.rb:158:in `call'
/usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/resource.rb:14:in `call'
/usr/lib/ruby/gems/1.8/gems/hammer_cli_import-0.10.6/lib/hammer_cli_import/base.rb:134:in `api_call'
===

Be sad.

Comment 6 Adam Price 2015-02-16 19:09:02 UTC
@grant if you actually want debug and verbose output, they need to come before the commands, because they're global flags.

so:
hammer -v -d import organization --csv-file /tmp/users.csv

Comment 7 Grant Gainey 2015-02-16 19:45:42 UTC
hammer-cli-import recognizes its own --debug and --verbose.  -v/-d control output from the underlying hammer-infrastructure.

Comment 8 Grant Gainey 2015-02-18 14:59:07 UTC
Fixing by special-casing org-create-call.

hammer-cli-import.github 59faf5b8cea6405c8d1d7ac07d1e94ce1ad09d49

Comment 9 Grant Gainey 2015-02-18 17:55:12 UTC
Modified by hammer-cli-import.github c106fa0a884567922124cba9108c7b88148dea34

Comment 13 Lukas Pramuk 2015-03-10 10:30:57 UTC
rubygem-hammer_cli_import-0.10.9.1
no such version on any compose?! ON_QA?

Comment 16 Lukas Pramuk 2015-03-10 14:16:41 UTC
VERIFIED. rubygem-hammer_cli_import-0.10.6.3-1.el6_6sat

# hammer import organization --verbose --csv-file /tmp/exports/users.csv --upload-manifests-from '/tmp/manifests''
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:68: warning: already initialized constant DEFAULT_LABEL_INDENT
Importing from /tmp/exports/users.csv
Creating new organization: RED HAT SATELLITE ENGINEERING
Uploading manifest /tmp/manifests/RED_HAT_SATELLITE_ENGINEERING.zip to org-id 3
Waiting for the task [4b8d6deb-4594-4913-affb-ef5e30b47bcb] .................
Creating new organization: Organization #2
Uploading manifest /tmp/manifests/Organization__2.zip to org-id 4
Waiting for the task [6f5e9af6-4a48-4f7c-829b-7b83f52f4714] ..........
Creating new organization: Organization #3
Uploading manifest /tmp/manifests/Organization__3.zip to org-id 5
Waiting for the task [3946507d-7ec2-48f4-8924-41d71e3d97d8] ..........
Summary
  Created 3 organizations.
  Uploaded 3 manifests.

Comment 17 Bryan Kearney 2015-08-11 13:23:09 UTC
This bug is slated to be released with Satellite 6.1.

Comment 18 errata-xmlrpc 2015-08-12 05:25:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592


Note You need to log in before you can comment on or make changes to this bug.