Bug 1131148 - [RFE] Mirror on Sync not working for puppet modules repositories
Summary: [RFE] Mirror on Sync not working for puppet modules repositories
Keywords:
Status: CLOSED DUPLICATE of bug 1294414
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.3
Hardware: x86_64
OS: Linux
medium
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On: 1131146 1175493
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2014-08-18 14:31 UTC by Peter Vreman
Modified: 2021-12-10 14:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-17 11:43:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker SAT-6846 0 None None None 2021-12-10 14:31:34 UTC
Red Hat Knowledge Base (Solution) 3004891 0 None None None 2017-04-18 09:57:43 UTC

Description Peter Vreman 2014-08-18 14:31:14 UTC
Description of problem:
There is no option to delete missing puppet modules that are not available anymore in the upstream repository. The result is that the number of puppet modules is growing, consuming space and becoming unmaintainable with the available_puppet_modules query.

This is especially important when Sat6 is connected to a CI environment where new builds of puppet modules are updated and changing very frequently. And with every build getting a higher version number and generated e.g. 50 modules the amount of puppet moduels grows very soon in 1000+ scales.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create puppet directory repository
2. Sync repository
3. Delete modules from repository
4. Sync repository again

Actual results:


Expected results:
Deleted modules are not available anymore


Additional info:
Related to pulp directory importer does not support remove_missing
 https://bugzilla.redhat.com/show_bug.cgi?id=1131146

Comment 1 RHEL Program Management 2014-08-18 14:33:06 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Bryan Kearney 2016-07-08 20:27:07 UTC
Per 6.3 planning, moving out non acked bugs to the backlog

Comment 5 Og Maciel 2017-04-17 18:38:32 UTC
This is an older bug which I do not envision being addressed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you.

Comment 6 Peter Vreman 2017-04-18 08:26:46 UTC
Re-open.

Our use-case is that the repository is creted by a CI system re-creating many RPMs and Puppet modules. The old modules are obsolete and can be deleted.
See below we can reach 10000+ puppet modules, but in fact only 50 are really in-use:

[crash/LI] root@li-lc-1017:~# sudo -u hoici hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml repository info --id=4412
ID:                 4412
Name:               HOIOS-7.3-ci-puppet
Label:              HOIOS-7_3-ci-puppet
Organization:       Hilti
Red Hat Repository: no
Content Type:       puppet
URL:                http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet
Publish Via HTTP:   yes
Published At:       http://li-lc-1017.hag.hilti.com/pulp/puppet/Hilti-HOIOS-HOIOS-7_3-ci-puppet/
Relative Path:      Hilti/Library/custom/HOIOS/HOIOS-7_3-ci-puppet
Product:
    ID:   40
    Name: HOIOS
GPG Key:

Sync:
    Status:         Success
    Last Sync Date: 6 days
Created:            2016/11/04 10:23:17
Updated:            2016/11/04 10:23:19
Content Counts:
    Puppet Modules: 12686

[crash/LI] root@li-lc-1017:~#


The feature to delete units during sync is supported during sin Pulp.
The integration with Katello is the only missing piece.

I now see that for RPM there is "Mirror on Sync", but for Puppet modules this is not supported, at least not visible in the UI.

Comment 7 Peter Vreman 2017-04-18 09:28:35 UTC
I think it is not an RFE, but a Bug. In the API response the option  "mirror_on_sync": true is set. But it is not used.

[crash/LI] root@li-lc-1017:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/4412 | jq .
{
  "content_type": "puppet",
  "docker_upstream_name": null,
  "mirror_on_sync": true,
  "unprotected": true,
  "full_path": "http://li-lc-1017.hag.hilti.com/pulp/puppet/Hilti-HOIOS-HOIOS-7_3-ci-puppet/",
  "checksum_type": null,
  "container_repository_name": null,
  "download_policy": null,
  "url": "http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet",
  "relative_path": "Hilti/Library/custom/HOIOS/HOIOS-7_3-ci-puppet",
  "major": null,
  "minor": null,
  "gpg_key_id": null,
  "content_id": "1478254998160",
  "content_view_version_id": 2,
  "library_instance_id": null,
  "product_type": "custom",
  "promoted": false,
  "ostree_branches": [],
  "organization": {
    "name": "Hilti",
    "label": "Hilti",
    "id": 3
  },
  "created_at": "2016-11-04 10:23:17 UTC",
  "updated_at": "2016-11-04 10:23:19 UTC",
  "id": 4412,
  "name": "HOIOS-7.3-ci-puppet",
  "label": "HOIOS-7_3-ci-puppet",
  "product": {
    "id": 40,
    "cp_id": "1476709488560",
    "name": "HOIOS",
    "sync_plan": [
      "name",
      "description",
      "sync_date",
      "interval",
      "next_sync"
    ]
  },
  "last_sync": {
    "id": "3cdfb1cc-7cff-4e73-bc8a-8d609207892d",
    "username": "hoici",
    "started_at": "2017-04-12 19:05:41 UTC",
    "ended_at": "2017-04-12 19:08:30 UTC",
    "state": "stopped",
    "result": "success",
    "progress": 1
  },
  "content_counts": {
    "ostree_branch": 0,
    "docker_manifest": 0,
    "docker_tag": 0,
    "rpm": 0,
    "package": 0,
    "package_group": 0,
    "erratum": 0,
    "puppet_module": 12686
  },
  "last_sync_words": "6 days",
  "gpg_key": null,
  "environment": {
    "id": 2
  },
  "permissions": {
    "deletable": true
  }
}


Below is the count (currently 57) Puppet modules available in the feed url:

[crash/LI] root@li-lc-1017:~# lftp -c 'o http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet ; ls' | grep -c .tar.gz
cd: received redirection to `http://fibsbamboodev.hag.hilti.com/hoici/release/HOIOS/7.3/HOIOS-7.3-ci/puppet/'
57

I expect that also the puppet repository in Satellite (Pulp) contains only those 57 puppet modules

Comment 9 Peter Vreman 2017-07-17 07:54:06 UTC
I think this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1294414

Comment 10 Bryan Kearney 2017-07-17 11:43:09 UTC
Thank you Peter. I am marking this as a duplicate per your suggestion.

*** This bug has been marked as a duplicate of bug 1294414 ***


Note You need to log in before you can comment on or make changes to this bug.