Bug 1637777

Summary: running forceful sync on an external capsule doesn't fix missing symbolics
Product: Red Hat Satellite Reporter: matt jia <mjia>
Component: Capsule - ContentAssignee: Ian Ballou <iballou>
Status: CLOSED ERRATA QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3.3CC: bbuckingham, daviddavis, iballou, jalviso, jsherril, mjia, ttereshc, vsedmik
Target Milestone: 6.7.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-14 13:23:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
task-export none

Description matt jia 2018-10-10 05:11:56 UTC
Description of problem:

Given that there are missing symbolics in a repo on an external capsule server,
running complete sync doesn't create those symbolics.

Version-Release number of selected component (if applicable):

6.3.3

How reproducible:

Easy

Steps to Reproduce:
1. sync a small repo(Red Hat Satellite Capsule 6.3 - Puppet 4 for RHEL 7 Server RPMs x86_64) to an external capsule. Make sure the repo is using immediate download policy 
2. rm a symbolic from capsule, for example:

rm -rf /var/lib/pulp/published/yum/master/yum_distributor/e3e783b1-7729-4248-92d9-1e399ae60417/1539145766.53/Packages/p/puppet-agent-oauth-0.5.1-2.el7sat.noarch.rpm

3. start a complete sync from Satellite UI

Actual results:

The deleted symbolic is not created.

Expected results:

The delted symbolic should be created

Additional info:

running below command doesn't bring back the symbolic either

pulp-admin -u admin -p $pulpAdminPassword rpm repo sync run --repo-id e3e783b1-7729-4248-92d9-1e399ae60417 --force-full

While examining the task details, we can see:

pulp-admin -u admin -p $pulpAdminPassword rpm repo list --repo-id e3e783b1-7729-4248-92d9-1e399ae60417  --details
---

  Distributor Type Id:  yum_distributor
  Id:                   e3e783b1-7729-4248-92d9-1e399ae60417
  Last Override Config: 

The Override Config should have  Force Full: True. The config is also missing when starting a forceful sync from satellite web UI. To workaround this issue, we have to do sth like

# Set the repository to be synced
repo_id = "e3e783b1-7729-4248-92d9-1e399ae60417"
capsule_api = Katello::CapsuleContent.new(SmartProxy.find(2)).pulp_server.resources
task_id = capsule_api.repository.publish(repo_id, repo_id, {override_config: {force_full: true}})

which is inconvenient.

Comment 1 David Davis 2018-10-10 17:18:55 UTC
Running a force_full sync will not fix missing symlinks in /var/lib/pulp/published/. What you instead need to run is:

    pulp-admin -vvvv rpm repo publish run --repo e3e783b1-7729-4248-92d9-1e399ae60417 --force-full

I don't think Satellite exposes this force_full publish functionality. Maybe it could or should though?

> The Override Config should have  Force Full: True.

I think this might be a bug. I will investigate further and open an upstream bug to fix this in Pulp.

Comment 2 David Davis 2018-10-10 21:22:28 UTC
I opened an upstream issue for the last override config not properly showing force_full.

https://pulp.plan.io/issues/4076

I am not linking them though as the issue is unrelated to this one.

I'm tempted to set the component on this issue to 'Katello' since Pulp does provide the ability to fix published symlinks.

Comment 3 matt jia 2018-10-10 22:55:10 UTC
>pulp-admin -vvvv rpm repo publish run --repo >e3e783b1-7729-4248-92d9-1e399ae60417 --force-full

>I don't think Satellite exposes this force_full publish functionality. Maybe it >could or should though?

Any drawback of not doing it? IMO, satellite should because we are asking for it to perform a forceful sync.

Comment 4 matt jia 2018-10-10 22:55:46 UTC
>Any drawback of not doing it?

I meant any drawback of doing it.

Comment 5 matt jia 2018-10-10 23:05:56 UTC
>Running a force_full sync will not fix missing symlinks in /var/lib>>/pulp/published/. What you instead need to run is:

    pulp-admin -vvvv rpm repo publish run --repo >e3e783b1-7729-4248-92d9-1e399ae60417 --force-full

David, I don't have any luck with this, see below 

pulp-admin -u admin -p $pulpAdminPassword -vvvv rpm repo publish run --help
Command: run
Description: triggers an immediate publish of a repository

Available Arguments:

  --bg      - if specified, the client process will end immediately (the task
              will continue to run on the server)
  --repo-id - (required) unique identifier; only alphanumeric, ., -, and _
              allowed

Options
  --force-full - forces to publish from scratch, even if no changes were
                 introduced since last publish

[root@mjia-capsule p]# pulp-admin -u admin -p $pulpAdminPassword -vvvv rpm repo publish run --repo-id e3e783b1-7729-4248-92d9-1e399ae60417 --force-full
+----------------------------------------------------------------------+
      Publishing Repository [e3e783b1-7729-4248-92d9-1e399ae60417]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Force Full:  True


2018-10-10 19:05:22,987 - DEBUG - sending POST request to /pulp/api/v2/tasks/search/
2018-10-10 19:05:23,101 - INFO - POST request to /pulp/api/v2/tasks/search/ with parameters {"criteria": {"filters": {"state": {"$nin": ["finished", "error", "canceled", "skipped"]}, "tags": {"$all": ["pulp:repository:e3e783b1-7729-4248-92d9-1e399ae60417", "pulp:action:publish"]}}}}
2018-10-10 19:05:23,102 - INFO - Response status : 200 

2018-10-10 19:05:23,102 - INFO - Response body :
 []

2018-10-10 19:05:23,102 - DEBUG - sending POST request to /pulp/api/v2/repositories/e3e783b1-7729-4248-92d9-1e399ae60417/actions/publish/
2018-10-10 19:05:23,217 - INFO - POST request to /pulp/api/v2/repositories/e3e783b1-7729-4248-92d9-1e399ae60417/actions/publish/ with parameters {"override_config": {"force_full": true}, "id": "yum_distributor"}
2018-10-10 19:05:23,217 - INFO - Response status : 202 

2018-10-10 19:05:23,220 - INFO - Response body :
 {
  "spawned_tasks": [
    {
      "_href": "/pulp/api/v2/tasks/75b4af54-36f4-463f-b38d-9f695a15e909/", 
      "task_id": "75b4af54-36f4-463f-b38d-9f695a15e909"
    }
  ], 
  "result": null, 
  "error": null
}

This command may be exited via ctrl+c without affecting the request.


2018-10-10 19:05:24,221 - DEBUG - sending GET request to /pulp/api/v2/tasks/75b4af54-36f4-463f-b38d-9f695a15e909/
2018-10-10 19:05:24,319 - INFO - GET request to /pulp/api/v2/tasks/75b4af54-36f4-463f-b38d-9f695a15e909/ with parameters None
2018-10-10 19:05:24,319 - INFO - Response status : 200 

2018-10-10 19:05:24,320 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.publish.publish", 
  "_href": "/pulp/api/v2/tasks/75b4af54-36f4-463f-b38d-9f695a15e909/", 
  "task_id": "75b4af54-36f4-463f-b38d-9f695a15e909", 
  "tags": [
    "pulp:repository:e3e783b1-7729-4248-92d9-1e399ae60417", 
    "pulp:action:publish"
  ], 
  "finish_time": "2018-10-10T23:05:23Z", 
  "_ns": "task_status", 
  "start_time": "2018-10-10T23:05:23Z", 
  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 527, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 438, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1097, in publish\n    dist = model.Distributor.objects.get_or_404(repo_id=repo_id, distributor_id=dist_id)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/db/querysets.py\", line 119, in get_or_404\n    raise pulp_exceptions.MissingResource(**kwargs)\nMissingResource: Missing resource(s): repo_id=e3e783b1-7729-4248-92d9-1e399ae60417, distributor_id=yum_distributor\n", 
  "spawned_tasks": [], 
  "progress_report": {}, 
  "queue": "reserved_resource_worker-2.gsslab.pek.redhat.com.dq", 
  "state": "error", 
  "worker_name": "reserved_resource_worker-2.gsslab.pek.redhat.com", 
  "result": null, 
  "error": {
    "code": "PLP0009", 
    "data": {
      "resources": {
        "repo_id": "e3e783b1-7729-4248-92d9-1e399ae60417", 
        "distributor_id": "yum_distributor"
      }
    }, 
    "description": "Missing resource(s): repo_id=e3e783b1-7729-4248-92d9-1e399ae60417, distributor_id=yum_distributor", 
    "sub_errors": []
  }, 
  "_id": {
    "$oid": "5bbe85b34c079f2d2eaa48ff"
  }, 
  "id": "5bbe85b34c079f2d2eaa48ff"
}


Task Failed

Missing resource(s): repo_id=e3e783b1-7729-4248-92d9-1e399ae60417,
distributor_id=yum_distributor

Comment 6 David Davis 2018-10-11 12:24:20 UTC
I forgot that Katello gives the distributor an id other than yum_distributor which won't work with the pulp-admin CLI. You can do it with the API though—although I am not sure it's easier than your method. First you'd have to find out the distributor id with "pulp-admin rpm repo list --details". Then you'd call POST /pulp/api/v2/repositories/e3e783b1-7729-4248-92d9-1e399ae60417/actions/publish/ 
with parameters {"override_config": {"force_full": true}, "id": "my_distributor_id"}.

> Any drawback of not doing it? IMO, satellite should because we are asking for it to perform a forceful sync.

Off the top of my head, I don't see any drawbacks to exposing force_full publishing in Katello or automatically perform a force_full publish after a force_full sync.

Comment 8 Justin Sherrill 2018-10-15 13:26:12 UTC
matt jia,

Would you be able to provide a tasks export from the satellite?   The way it works is the foreman-task should be kicking off a 2nd publish with force_full set to true.  I'd love to see the task export to confirm if it is doing that or not.

Thanks,

Justin

Comment 9 matt jia 2018-10-16 00:15:32 UTC
Created attachment 1494193 [details]
task-export

Comment 11 Tanya Tereshchenko 2020-01-03 17:06:12 UTC
Pulp provides a way to perform force_full publish which should fix the broken symlinks.

The issue is potentially on the Katello side, or at least investigation should be done there first to see what calls are made to Pulp. Should we move it to a different component?
Maybe it's worth closing this bug completely since the report seems to be 1+ year old.

Comment 12 Justin Sherrill 2020-01-03 17:46:32 UTC
It looks like this was fixed in Satellite 6.6 as part of https://bugzilla.redhat.com/show_bug.cgi?id=1731060  (although the description is quite different).   Will need to test to confirm.

Comment 13 Ian Ballou 2020-01-14 16:37:09 UTC
I've tested this in Satellite 6.6 and can confirm that the symbolic link is recreated after doing a full sync.

Comment 17 errata-xmlrpc 2020-04-14 13:23:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454