Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1743776 - Error while deleting the content view version.
Summary: Error while deleting the content view version.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.5.0
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: 6.8.0
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-20 16:14 UTC by Ganesh Payelkar
Modified: 2024-10-01 16:19 UTC (History)
18 users (show)

Fixed In Version: pulp-rpm-2.21.3-1
Doc Type: Known Issue
Doc Text:
Error while deleting the content view version
Clone Of:
: 1851136 (view as bug list)
Environment:
Last Closed: 2020-10-27 12:59:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 2826 0 Normal CLOSED - CURRENTRELEASE Repository deletion fails with no such file for a repositories listing file 2020-07-14 11:07:19 UTC
Red Hat Knowledge Base (Solution) 5277541 0 None None None 2020-07-30 22:50:55 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 12:59:20 UTC

Description Ganesh Payelkar 2019-08-20 16:14:17 UTC
Description of problem:

Getting the below error when trying to delete a content-view version. Try to clean up the content-view version through API, and hit this issue randomly.

Need to manually skip/resume these tasks to successfully delete the content-view version but getting stuck on below error.


Version-Release number of selected component (if applicable):
6.4


Actual results:

pulp_tasks:
- exception:
  task_type: pulp.server.tasks.repository.delete
  _href: "/pulp/api/v2/tasks/e7eaf752-0e62-4292-b200-28b475002962/"
  task_id: e7eaf752-0e62-4292-b200-28b475002962
  tags:
  - pulp:repository:1-RHEL_7-v785_0-b5b43a8f-3598-4a1b-9400-5625d6f970af
  - pulp:action:delete
  finish_time: '2019-08-16T06:15:46Z'
  _ns: task_status
  start_time: '2019-08-16T06:15:46Z'
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
        R = retval = fun(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
        return super(Task, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
        return super(PulpTask, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
        return self.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 559, in delete
        raise pe
    PulpExecutionException: Pulp exception occurred: PulpExecutionException
  spawned_tasks: []
  progress_report: {}
  queue: reserved_resource_worker-1@satellite
  state: error
  worker_name: reserved_resource_worker-1@satellite
  result:
  error:
    code: PLP0000
    data: {}
    description: 'Pulp exception occurred: PulpExecutionException'
    sub_errors:
    - code: PLP0000
      data: {}
      description: "[Errno 2] No such file or directory: '/var/lib/pulp/published/yum/https/repos/redhat/content_views/RHEL_7/78.0/content/dist/rhel/server/7/7Server/x86_64/sat-tools/listing'"
      sub_errors: []
  _id:
    "$oid": 5d564a128e667f6b36e52fc7
  id: 5d564a128e667f6b36e52fc7
poll_attempts:
  total: 1
  failed: 1



Expected results:

content-view has to get deleted without any error and without going into a paused state. 


Additional info:

file - /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   try:
        # Remove all importers and distributors from the repo. This is likely already done by the
        # calls to other methods in this manager, but in case those failed we still want to attempt
        # to keep the database clean.
        model.Distributor.objects(repo_id=repo_id).delete()
        model.Importer.objects(repo_id=repo_id).delete()
        RepoSyncResult.get_collection().remove({'repo_id': repo_id})
        RepoPublishResult.get_collection().remove({'repo_id': repo_id})
        RepoContentUnit.get_collection().remove({'repo_id': repo_id})
    except Exception, e:
        msg = _('Error updating one or more database collections while removing repo [%(r)s]')
        msg = msg % {'r': repo_id}
        _logger.exception(msg)
        error_tuples.append(e)

    # remove the repo from any groups it was a member of
    group_manager = manager_factory.repo_group_manager()
    group_manager.remove_repo_from_groups(repo_id)

    if len(error_tuples) > 0:
        pe = pulp_exceptions.PulpExecutionException()
        pe.child_exceptions = error_tuples
        raise pe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From the code comments, I can see that it is trying to re-delete the distributor and content but it is obvious that if it is trying to re-delete it will not find the file "/var/lib/pulp/published/yum/https/repos/redhat/content_views/RHEL_7/78.0/content/dist/rhel/server/7/7Server/x86_64/sat-tools/listing".

Comment 3 Dylan Gross 2019-08-22 21:18:13 UTC
Changing bug from affected version 6.4.2 to 6.5.0.  (It was really encountered on 6.5.2, but that's not a valid dropdown selection yet)

Comment 7 Tanya Tereshchenko 2020-03-19 20:49:35 UTC
The exception raised as part of aggregated set of errors, so it would be helpful to see pulp logs because all details are there, unfortunately. And they should point to a spot where it happens.


> From the code comments, I can see that it is trying to re-delete the distributor and content but it is obvious that if it is trying to re-delete it will not find the file >"/var/lib/pulp/published/yum/https/repos/redhat/content_views/RHEL_7/78.0/content/dist/rhel/server/7/7Server/x86_64/sat-tools/listing".

I don't think it's an issue, the attempt there is to clean database and not the filesystem.


I can't reproduce the issue and the only potential spot I found is here.
https://github.com/pulp/pulp_rpm/compare/2-master...goosemania:issue2826.patch

If anyone can try the patch and/or reproduce and share the logs, it would be helpful.

Comment 8 pulp-infra@redhat.com 2020-03-23 12:12:14 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2020-03-23 12:12:16 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 14 Tanya Tereshchenko 2020-06-15 17:01:53 UTC
Hi Dylan,

Thanks for this!
I'm glad there is a place to reproduce the issue and test the patch. I was looking for it. Sorry to hear that the customer needed to dig into the code.
With my blind search, I found the second spot with my initial patch from comment#7 but not the first one. Thanks for figuring it out.

The suggested patch by the customer has some issues. Depending on the reasons for the file or directory removal to fail, code should react differently.
We can safely ignore only the case when they no longer exist, any other OSError is not safe to ignore and it would be better if failure is not ignored but propagated, e.g. permission problem.

`If` statements are not safe to use due to race condition nature of the problem, so I added try/except for the first spot you found as well. 
Here is the suggested patch to test https://patch-diff.githubusercontent.com/raw/pulp/pulp_rpm/pull/1748.patch

If it works, we will include it in the upcoming release.

Comment 15 pulp-infra@redhat.com 2020-06-15 17:12:59 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2020-06-23 20:06:01 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2020-06-23 21:05:34 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 20 pulp-infra@redhat.com 2020-07-14 11:07:20 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2020-09-08 20:09:13 UTC
Requesting needsinfo from upstream developer ttereshc because the 'FailedQA' flag is set.

Comment 23 David Davis 2020-09-09 18:17:41 UTC
This should be fixed in pulp-rpm-plugins-2.21.3-1

Comment 24 Lai 2020-09-15 15:22:57 UTC
Steps to reproduce.

1. Enabling a bunch of red hat repos (mine was around 10)
   1a (optional) - sync repos
2. Create a cv and add all the repos from 1 to it
3. Publish repo 2 - 3 times
4. Delete one cv version
5. Check task page
6. Check distributor.py for patch


Expected:
5. Task should return no error
6. The code should be the same as mentioned in comment #14

Actual:
5. Task returns no errors for deleting cv version
6. The code is the same as mentioned in comment #14

Also checked the pulp version:
pulp-rpm-plugins-2.21.3-2.el7sat.noarch

Verified on 6.8.0 snap 15

Comment 27 errata-xmlrpc 2020-10-27 12:59:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366


Note You need to log in before you can comment on or make changes to this bug.