Bug 1430879

Summary: Support verify content sync on bulk products
Product: Red Hat Satellite Reporter: Justin Sherrill <jsherril>
Component: Content ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: high    
Version: 6.3.0CC: bbuckingham, jcallaha, jsherril, katello-qa-list, mmccune, sghai, zhunting
Target Milestone: UnspecifiedKeywords: PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1427618
: 1530485 (view as bug list) Environment:
Last Closed: 2018-02-21 16:54:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1223023, 1427618    
Bug Blocks: 1530485, 1686971    
Attachments:
Description Flags
advanced sync with 3 sync options on product page none

Comment 1 Justin Sherrill 2017-03-09 18:49:52 UTC
This is for adding UI/API/CLI for bulk product force and verify syncs

Comment 3 Satellite Program 2017-03-09 19:10:32 UTC
Upstream bug assigned to jsherril

Comment 4 Satellite Program 2017-03-16 20:11:01 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18844 has been resolved.

Comment 5 Sachin Ghai 2017-12-28 14:13:22 UTC
Verified with sat6.3 snap30

I can see the option to verify content sync on bulk products using UI. 3 new options have been added on selecting "Advanced Sync" from bulk action drop down.

a) optimized sync b) complete sync c) validate content sync


Please see the attachment

Comment 6 Sachin Ghai 2017-12-28 14:14:31 UTC
Created attachment 1373244 [details]
advanced sync with 3 sync options on product page

Comment 7 Sachin Ghai 2017-12-28 14:16:52 UTC
Running "validate content sync" on on_demand sync policy yum repos throws error:

 RuntimeError: No syncable repositories found for selected products and options. 

This option is valid only for "immediate" and backround sync policy yum repos.

Comment 8 Sachin Ghai 2017-12-28 14:23:20 UTC
From UI, sync on bulk product is working fine though I feel we can improve on two things:

a) error message as mentioned in comment7 for "validate content sync" for on_demand sync policy yum repos is a bit misleading. I think we should update it like:

Validate content sync operation is not permitted for on_demand sync policy yum repos

I'll file another bug for it.

b) when we run sync on bulk products, as soon as we submit the form, UI should navigate from product page to sync status page or atleast unselect all selected repos. ( will file seperate issue)


I'll test the same through cli and api as well.

Comment 9 Sachin Ghai 2017-12-29 11:15:20 UTC
I can see following api call only (katello/api/products/40/sync) for bulk product sync.

 ~]# curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X POST 'https://dell-pet410-04.khw.lab.eng.bos.redhat.com/katello/api/products/40/sync'
HTTP/1.1 202 Accepted
 {"id":"19acab7a-9847-45d5-85a4-1e45135e7904","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2017-12-29 11:09:12 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5],"target_class":"Katello::Repository","args":[],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}


There is no hammer command for bulk product sync.

Subcommands:
 create                        Create a product
 delete                        Destroy a product
 info                          Show a product
 list                          List products
 remove-sync-plan              Delete assignment sync plan and product.
 set-sync-plan                 Assign sync plan to product.
 synchronize                   Sync all repositories for a product
 update                        Updates a product



@Justin: I can see UI changes around bulk products sync as stated above but no such cmds in hammer ?

bz description we should have support from CLI and api as well. whats expected here ? please suggest.

Comment 10 Justin Sherrill 2017-12-29 15:44:13 UTC
Sachin, 

The bulk product api is actually:

PUT /products/bulk/sync/

with a body of:

{
  "ids': [1,2,3],
  "validate_contents": true
}

The api you mentioned is the single product sync api.


To answer your question about the cli, you are correct, this is not in the cli currently.  In fact there are no bulk actions available via the cli for products at all (and i don't think they ever existed).  

My suggestion would be to open a new bug/clone this bug to create hammer commands for all bulk product actions (sync, delete, update sync plans).  We could fail_qa this bug, but i don't see this being done for 6.3 (but maybe the triage team will disagree).

Comment 11 Sachin Ghai 2018-01-02 11:18:01 UTC
Thank you Justin for your reply. I see the bulk product api and can sync multiple products w/ validate_content option.


 ~]#  curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X PUT -d '{"ids": [40,41], "validate_contents": "true"}' 'https://SAT_SERVER/katello/api/products/bulk/sync'
HTTP/1.1 202 Accepted
Status: 202 Accepted
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8

  {"id":"7153215c-f441-4470-8823-e4683c97e6ed","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2018-01-02 11:06:18 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5,6],"target_class":"Katello::Repository","args":[null,{"skip_metadata_check":false,"validate_contents":true}],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}



I'll open a new bug to add hammer commands for bulk product actions. thanks.

Comment 12 Sachin Ghai 2018-01-03 06:33:29 UTC
following bz is opened to add support in hammer for bulk product actions:

https://bugzilla.redhat.com/show_bug.cgi?id=1530485


For issue b) in comment8: https://bugzilla.redhat.com/show_bug.cgi?id=1529677

Comment 13 Sachin Ghai 2018-01-03 06:37:55 UTC
Bulk product sync through api using flag skip_metadata_check:

# curl -i -k -u admin:changeme  -H "Content-Type: application/json" -X PUT -d '{"ids": [40,41], "skip_metadata_check": "true"}' 'https://SAT_SERVER/katello/api/products/bulk/sync'
HTTP/1.1 202 Accepted
Date: Wed, 03 Jan 2018 06:30:07 GMT
----
Status: 202 Accepted
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8

  {"id":"3366646a-535d-4751-806c-29fffa68dab8","label":"Actions::BulkAction","pending":true,"username":"admin","started_at":"2018-01-03 06:30:07 UTC","ended_at":null,"state":"planned","result":"pending","progress":0.0,"input":{"action_class":"Actions::Katello::Repository::Sync","target_ids":[5,6],"target_class":"Katello::Repository","args":[null,{"skip_metadata_check":true,"validate_contents":false}],"current_user_id":3},"output":{},"humanized":{"action":"Bulk action","input":null,"output":null,"errors":[]},"cli_example":null}


from production.log:
==================

 [I] Started POST "/katello/api/v2/repositories/sync_complete?token=2SWS4odpCFAqfYmMgtaaBmVMP77NwWLK" for 10.16.185.162 at 2018-01-03 01:31:41 -0500
2018-01-03 01:31:41 3bf54494 [app] [I] Processing by Katello::Api::V2::RepositoriesController#sync_complete as */*
2018-01-03 01:31:41 3bf54494 [app] [I]   Parameters: {"call_report"=>"[FILTERED]", "event_type"=>"repo.sync.finish", "payload"=>{"importer_id"=>"yum_importer", "exception"=>nil, "repo_id"=>"21b3569b-cd00-4831-a102-3cd869d313c6", "traceback"=>nil, "started"=>"2018-01-03T06:30:15Z", "_ns"=>"repo_sync_results", "completed"=>"2018-01-03T06:31:39Z", "importer_type_id"=>"yum_importer", "error_message"=>nil, "summary"=>{"content"=>{"state"=>"FINISHED"}, "comps"=>{"state"=>"FINISHED"}, "purge_duplicates"=>{"state"=>"FINISHED"}, "distribution"=>{"state"=>"FINISHED"}, "errata"=>{"state"=>"FINISHED"}, "metadata"=>{"state"=>"FINISHED"}}, "added_count"=>0, "result"=>"success", "updated_count"=>2, "details"=>{"content"=>{"size_total"=>0, "items_left"=>0, "items_total"=>0, "state"=>"FINISHED", "size_left"=>0, "details"=>{"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0}, "error_details"=>[]}, "comps"=>{"state"=>"FINISHED"}, "purge_duplicates"=>{"state"=>"FINISHED"}, "distribution"=>{"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "items_left"=>0}, "errata"=>{"state"=>"FINISHED"}, "metadata"=>{"state"=>"FINISHED"}}, "id"=>"5a4c78cbef0f7a33c29812a1", "removed_count"=>0}, "token"=>"2SWS4odpCFAqfYmMgtaaBmVMP77NwWLK", "api_version"=>"v2", "repository"=>{}}

Comment 14 Sachin Ghai 2018-01-03 09:19:28 UTC
I corrupted a package and later resynced the repo from UI by selecting option "validate content sync" and package was resycned. I confirmed it through md5sum


[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# ll
total 64
-rw-r--r--. 1 apache apache 61552 Dec 22 07:29 libgudev1-147-2.46.el6.x86_64.rpm
[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# pwd
/var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8


Checked the md5sum before corrupting the package:
=====================================================
[root@dell-pet410-04 fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8]# md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm 
d0323018ad8675c16dd94232dcf719b2  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
 

corrupted the package
=======================
echo "FOO" >> /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm

md5sum changed after corrupting the package
===========================================

md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
5fde6817f5ac9c40deba6fe385e436e3  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm


resync the repo using validate content sync option and checked the md5sum and it shows the same md5sum which was set originally.

# md5sum /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm
d0323018ad8675c16dd94232dcf719b2  /var/lib/pulp/content/units/rpm/fe/fe1f2946d57d9b26c9ea16e0b777ceb47287bd440e974909db071350ccd8e8/libgudev1-147-2.46.el6.x86_64.rpm

Comment 15 Satellite Program 2018-02-21 16:54:37 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336