Bug 1131148

Summary: [RFE] Mirror on Sync not working for puppet modules repositories
Product: Red Hat Satellite Reporter: Peter Vreman <peter.vreman>
Component: Content ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0.3CC: bkearney, brubisch, omaciel
Target Milestone: UnspecifiedKeywords: FutureFeature, Reopened, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-17 11:43:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1131146, 1175493    
Bug Blocks: 1122832    

Description Peter Vreman 2014-08-18 14:31:14 UTC
Description of problem:
There is no option to delete missing puppet modules that are not available anymore in the upstream repository. The result is that the number of puppet modules is growing, consuming space and becoming unmaintainable with the available_puppet_modules query.

This is especially important when Sat6 is connected to a CI environment where new builds of puppet modules are updated and changing very frequently. And with every build getting a higher version number and generated e.g. 50 modules the amount of puppet moduels grows very soon in 1000+ scales.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create puppet directory repository
2. Sync repository
3. Delete modules from repository
4. Sync repository again

Actual results:


Expected results:
Deleted modules are not available anymore


Additional info:
Related to pulp directory importer does not support remove_missing
 https://bugzilla.redhat.com/show_bug.cgi?id=1131146

Comment 1 RHEL Program Management 2014-08-18 14:33:06 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Bryan Kearney 2016-07-08 20:27:07 UTC
Per 6.3 planning, moving out non acked bugs to the backlog

Comment 5 Og Maciel 2017-04-17 18:38:32 UTC
This is an older bug which I do not envision being addressed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you.

Comment 6 Peter Vreman 2017-04-18 08:26:46 UTC
Re-open.

Our use-case is that the repository is creted by a CI system re-creating many RPMs and Puppet modules. The old modules are obsolete and can be deleted.
See below we can reach 10000+ puppet modules, but in fact only 50 are really in-use:

[crash/LI] root@li-lc-1017:~# sudo -u hoici hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml repository info --id=4412
ID:                 4412
Name:               HOIOS-7.3-ci-puppet
Label:              HOIOS-7_3-ci-puppet
Organization:       Hilti
Red Hat Repository: no
Content Type:       puppet
URL:                http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet
Publish Via HTTP:   yes
Published At:       http://li-lc-1017.hag.hilti.com/pulp/puppet/Hilti-HOIOS-HOIOS-7_3-ci-puppet/
Relative Path:      Hilti/Library/custom/HOIOS/HOIOS-7_3-ci-puppet
Product:
    ID:   40
    Name: HOIOS
GPG Key:

Sync:
    Status:         Success
    Last Sync Date: 6 days
Created:            2016/11/04 10:23:17
Updated:            2016/11/04 10:23:19
Content Counts:
    Puppet Modules: 12686

[crash/LI] root@li-lc-1017:~#


The feature to delete units during sync is supported during sin Pulp.
The integration with Katello is the only missing piece.

I now see that for RPM there is "Mirror on Sync", but for Puppet modules this is not supported, at least not visible in the UI.

Comment 7 Peter Vreman 2017-04-18 09:28:35 UTC
I think it is not an RFE, but a Bug. In the API response the option  "mirror_on_sync": true is set. But it is not used.

[crash/LI] root@li-lc-1017:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/4412 | jq .
{
  "content_type": "puppet",
  "docker_upstream_name": null,
  "mirror_on_sync": true,
  "unprotected": true,
  "full_path": "http://li-lc-1017.hag.hilti.com/pulp/puppet/Hilti-HOIOS-HOIOS-7_3-ci-puppet/",
  "checksum_type": null,
  "container_repository_name": null,
  "download_policy": null,
  "url": "http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet",
  "relative_path": "Hilti/Library/custom/HOIOS/HOIOS-7_3-ci-puppet",
  "major": null,
  "minor": null,
  "gpg_key_id": null,
  "content_id": "1478254998160",
  "content_view_version_id": 2,
  "library_instance_id": null,
  "product_type": "custom",
  "promoted": false,
  "ostree_branches": [],
  "organization": {
    "name": "Hilti",
    "label": "Hilti",
    "id": 3
  },
  "created_at": "2016-11-04 10:23:17 UTC",
  "updated_at": "2016-11-04 10:23:19 UTC",
  "id": 4412,
  "name": "HOIOS-7.3-ci-puppet",
  "label": "HOIOS-7_3-ci-puppet",
  "product": {
    "id": 40,
    "cp_id": "1476709488560",
    "name": "HOIOS",
    "sync_plan": [
      "name",
      "description",
      "sync_date",
      "interval",
      "next_sync"
    ]
  },
  "last_sync": {
    "id": "3cdfb1cc-7cff-4e73-bc8a-8d609207892d",
    "username": "hoici",
    "started_at": "2017-04-12 19:05:41 UTC",
    "ended_at": "2017-04-12 19:08:30 UTC",
    "state": "stopped",
    "result": "success",
    "progress": 1
  },
  "content_counts": {
    "ostree_branch": 0,
    "docker_manifest": 0,
    "docker_tag": 0,
    "rpm": 0,
    "package": 0,
    "package_group": 0,
    "erratum": 0,
    "puppet_module": 12686
  },
  "last_sync_words": "6 days",
  "gpg_key": null,
  "environment": {
    "id": 2
  },
  "permissions": {
    "deletable": true
  }
}


Below is the count (currently 57) Puppet modules available in the feed url:

[crash/LI] root@li-lc-1017:~# lftp -c 'o http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet ; ls' | grep -c .tar.gz
cd: received redirection to `http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet/'
57

I expect that also the puppet repository in Satellite (Pulp) contains only those 57 puppet modules

Comment 9 Peter Vreman 2017-07-17 07:54:06 UTC
I think this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1294414

Comment 10 Bryan Kearney 2017-07-17 11:43:09 UTC
Thank you Peter. I am marking this as a duplicate per your suggestion.

*** This bug has been marked as a duplicate of bug 1294414 ***