This is for adding UI/API/CLI for bulk product force and verify syncs

Upstream bug assigned to jsherril

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18844 has been resolved.

Verified with sat6.3 snap30

I can see the option to verify content sync on bulk products using UI. 3 new options have been added on selecting "Advanced Sync" from bulk action drop down.

a) optimized sync b) complete sync c) validate content sync


Please see the attachment

Created attachment 1373244 [details]
advanced sync with 3 sync options on product page

Running "validate content sync" on on_demand sync policy yum repos throws error:

 RuntimeError: No syncable repositories found for selected products and options. 

This option is valid only for "immediate" and backround sync policy yum repos.

From UI, sync on bulk product is working fine though I feel we can improve on two things:

a) error message as mentioned in comment7 for "validate content sync" for on_demand sync policy yum repos is a bit misleading. I think we should update it like:

Validate content sync operation is not permitted for on_demand sync policy yum repos

I'll file another bug for it.

b) when we run sync on bulk products, as soon as we submit the form, UI should navigate from product page to sync status page or atleast unselect all selected repos. ( will file seperate issue)


I'll test the same through cli and api as well.

I can see following api call only (katello/api/products/40/sync) for bulk product sync.

 ~]# curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X POST 'https://dell-pet410-04.khw.lab.eng.bos.redhat.com/katello/api/products/40/sync'
HTTP/1.1 202 Accepted
 {"id":"19acab7a-9847-45d5-85a4-1e45135e7904","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2017-12-29 11:09:12 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5],"target_class":"Katello::Repository","args":[],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}


There is no hammer command for bulk product sync.

Subcommands:
 create                        Create a product
 delete                        Destroy a product
 info                          Show a product
 list                          List products
 remove-sync-plan              Delete assignment sync plan and product.
 set-sync-plan                 Assign sync plan to product.
 synchronize                   Sync all repositories for a product
 update                        Updates a product



@Justin: I can see UI changes around bulk products sync as stated above but no such cmds in hammer ?

bz description we should have support from CLI and api as well. whats expected here ? please suggest.

Sachin, 

The bulk product api is actually:

PUT /products/bulk/sync/

with a body of:

{
  "ids': [1,2,3],
  "validate_contents": true
}

The api you mentioned is the single product sync api.


To answer your question about the cli, you are correct, this is not in the cli currently.  In fact there are no bulk actions available via the cli for products at all (and i don't think they ever existed).  

My suggestion would be to open a new bug/clone this bug to create hammer commands for all bulk product actions (sync, delete, update sync plans).  We could fail_qa this bug, but i don't see this being done for 6.3 (but maybe the triage team will disagree).

Thank you Justin for your reply. I see the bulk product api and can sync multiple products w/ validate_content option.


 ~]#  curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X PUT -d '{"ids": [40,41], "validate_contents": "true"}' 'https://SAT_SERVER/katello/api/products/bulk/sync'
HTTP/1.1 202 Accepted
Status: 202 Accepted
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8

  {"id":"7153215c-f441-4470-8823-e4683c97e6ed","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2018-01-02 11:06:18 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5,6],"target_class":"Katello::Repository","args":[null,{"skip_metadata_check":false,"validate_contents":true}],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}



I'll open a new bug to add hammer commands for bulk product actions. thanks.

following bz is opened to add support in hammer for bulk product actions:

https://bugzilla.redhat.com/show_bug.cgi?id=1530485


For issue b) in comment8: https://bugzilla.redhat.com/show_bug.cgi?id=1529677

Bulk product sync through api using flag skip_metadata_check:

# curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X PUT -d '{"ids": [40,41], "skip_metadata_check": "true"}' 'https://SAT_SERVER/katello/api/products/bulk/sync'
HTTP/1.1 202 Accepted
Date: Wed, 03 Jan 2018 06:30:07 GMT
----
Status: 202 Accepted
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8

  {"id":"3366646a-535d-4751-806c-29fffa68dab8","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2018-01-03 06:30:07 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5,6],"target_class":"Katello::Repository","args":[null,{"skip_metadata_check":true,"validate_contents":false}],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}


from production.log:
==================

 [I] Started POST "/katello/api/v2/repositories/sync_complete?token=2SWS4odpCFAqfYmMgtaaBmVMP77NwWLK" for 10.16.185.162 at 2018-01-03 01:31:41 -0500
2018-01-03 01:31:41 3bf54494 [app] [I] Processing by Katello::Api::V2::RepositoriesController#sync_complete as */*
2018-01-03 01:31:41 3bf54494 [app] [I]   Parameters: {"call_report"=>"[FILTERED]", "event_type"=>"repo.sync.finish", "payload"=>{"importer_id"=>"yum_importer", "exception"=>nil, "repo_id"=>"21b3569b-cd00-4831-a102-3cd869d313c6", "traceback"=>nil, "started"=>"2018-01-03T06:30:15Z", "_ns"=>"repo_sync_results", "completed"=>"2018-01-03T06:31:39Z", "importer_type_id"=>"yum_importer", "error_message"=>nil, "summary"=>{"content"=>{"state"=>"FINISHED"}, "comps"=>{"state"=>"FINISHED"}, "purge_duplicates"=>{"state"=>"FINISHED"}, "distribution"=>{"state"=>"FINISHED"}, "errata"=>{"state"=>"FINISHED"}, "metadata"=>{"state"=>"FINISHED"}}, "added_count"=>0, "result"=>"success", "updated_count"=>2, "details"=>{"content"=>{"size_total"=>0, "items_left"=>0, "items_total"=>0, "state"=>"FINISHED", "size_left"=>0, "details"=>{"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0}, "error_details"=>[]}, "comps"=>{"state"=>"FINISHED"}, "purge_duplicates"=>{"state"=>"FINISHED"}, "distribution"=>{"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "items_left"=>0}, "errata"=>{"state"=>"FINISHED"}, "metadata"=>{"state"=>"FINISHED"}}, "id"=>"5a4c78cbef0f7a33c29812a1", "removed_count"=>0}, "token"=>"2SWS4odpCFAqfYmMgtaaBmVMP77NwWLK", "api_version"=>"v2", "repository"=>{}}

I corrupted a package and later resynced the repo from UI by selecting option "validate content sync" and package was resycned. I confirmed it through md5sum


[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# ll
total 64
-rw-r--r--. 1 apache apache 61552 Dec 22 07:29 libgudev1-147-2.46.el6.x86_64.rpm
[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# pwd
/var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8


Checked the md5sum before corrupting the package:
=====================================================
[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm 
d0323018ad8675c16dd94232dcf719b2  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
 

corrupted the package
=======================
echo "FOO" >> /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm

md5sum changed after corrupting the package
===========================================

md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
5fde6817f5ac9c40deba6fe385e436e3  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm


resync the repo using validate content sync option and checked the md5sum and it shows the same md5sum which was set originally.

# md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
d0323018ad8675c16dd94232dcf719b2  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336