Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1993773 - Synchronization of "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" fails with an unhelpful error "Pulp task error"
Summary: Synchronization of "Red Hat Enterprise Linux 7 Server - Extended Update Suppo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: 6.10.0
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
: 1996106 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-16 05:51 UTC by Sayan Das
Modified: 2021-11-16 14:13 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 14:13:06 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 9491 0 High CLOSED - CURRENTRELEASE Backport #9301 "Users running into timeouts with error messages that are unhelpful" to 3.14.z 2021-10-06 14:07:45 UTC
Red Hat Product Errata RHSA-2021:4702 0 None None None 2021-11-16 14:13:18 UTC

Description Sayan Das 2021-08-16 05:51:03 UTC 
Description of problem:

Attempted to sync RHEL 7.7 EUS repo twice and both time it failed with some very unhelpful error i.e. "Pulp task error".

Version-Release number of selected component (if applicable):
- Satellite 6.10.0 [ latest snap satellite-6.10.0-0.6.beta.el7sat.noarch ]

How reproducible:
- Always [ At least two consecutive attempts failed ]

Steps to Reproduce:
1. Install satellite 6.10 and import a manifest
2. Enable few small and big RHEL repos along with "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7"
3. Sync them
4. Try to resync "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7"

Actual results:

Sync of "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" will always fail with following error in UI.

"Pulp task error"


And folloing traceback in syslog :


Aug 16 10:57:52 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]:  - - [16/Aug/2021:05:27:52 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 1035 "-" "OpenAPI-Generator/3.14.1/ruby"
Aug 16 10:58:08 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]:  - - [16/Aug/2021:05:28:08 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 1035 "-" "OpenAPI-Generator/3.14.1/ruby"
Aug 16 10:58:22 satellite pulpcore-worker-1: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: pulpcore.tasking.pulpcore_worker:INFO: Task 7bb636dc-3b2a-4f7f-99b2-56f6c0164405 failed ()
Aug 16 10:58:22 satellite pulpcore-worker-1: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
Aug 16 10:58:22 satellite pulpcore-worker-1: result = func(*args, **kwargs)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 427, in synchronize
Aug 16 10:58:22 satellite pulpcore-worker-1: version = dv.create()
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Aug 16 10:58:22 satellite pulpcore-worker-1: loop.run_until_complete(pipeline)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 16 10:58:22 satellite pulpcore-worker-1: return future.result()
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Aug 16 10:58:22 satellite pulpcore-worker-1: await asyncio.gather(*futures)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 16 10:58:22 satellite pulpcore-worker-1: await self.run()
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 659, in run
Aug 16 10:58:22 satellite pulpcore-worker-1: name, location_href, result = await future
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/asyncio/tasks.py", line 458, in _wait_for_one
Aug 16 10:58:22 satellite pulpcore-worker-1: return f.result()  # May raise f.exception().
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 631, in run_repomdrecord_download
Aug 16 10:58:22 satellite pulpcore-worker-1: result = await downloader.run()
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
Aug 16 10:58:22 satellite pulpcore-worker-1: return await download_wrapper()
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
Aug 16 10:58:22 satellite pulpcore-worker-1: ret = await target(*args, **kwargs)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
Aug 16 10:58:22 satellite pulpcore-worker-1: return await self._run(extra_data=extra_data)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 95, in _run
Aug 16 10:58:22 satellite pulpcore-worker-1: to_return = await self._handle_response(response)
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 210, in _handle_response
Aug 16 10:58:22 satellite pulpcore-worker-1: chunk = await response.content.read(1048576)  # 1 megabyte
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
Aug 16 10:58:22 satellite pulpcore-worker-1: await self._wait("read")
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
Aug 16 10:58:22 satellite pulpcore-worker-1: await waiter
Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/helpers.py", line 656, in __exit__
Aug 16 10:58:22 satellite pulpcore-worker-1: raise asyncio.TimeoutError from None
Aug 16 10:58:24 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]:  - - [16/Aug/2021:05:28:24 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 3393 "-" "OpenAPI-Generator/3.14.1/ruby"

Expected results:

1. The repo should be synced properly.

2. If the repo sync is failing, an understandable Error message or traceback should be printed in foreman-tasks or syslogs so that we can come to know what is the exact issue and why it failed.


Additional info:

- Task export for both of the failed sync attempts and /var/log/messages from satellite server will be uploaded on the case.

- It could be related to a low memory situation but the error is not clear at all and pulp2 can sync this repo without any issues.

- NOTE: All services in satellite is running fine and "hammer ping" shows OK for all components.
Comment 6 Sayan Das 2021-08-19 09:43:22 UTC 
I have checked it again on a 4vCPU\20 GB ram satellite 6.10, The issue remains same.

If I try to sync EUS 7.7 with any other big repos at once, all repos will get sync except EUS 7.7 and the point of failure is exactly the same what was reported in the Description.
Comment 9 Tanya Tereshchenko 2021-08-23 20:06:25 UTC 
*** Bug 1996106 has been marked as a duplicate of this bug. ***
Comment 10 pulp-infra@redhat.com 2021-09-08 16:11:29 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2021-09-08 16:11:30 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 12 pulp-infra@redhat.com 2021-09-12 23:08:07 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 14 pulp-infra@redhat.com 2021-09-27 15:08:41 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 15 Dennis Kliban 2021-09-28 21:02:35 UTC 
It's possible that this bug is related to a rate limiting caused by the DNS servers. Could you please try to reproduce when the DNS is configured to use 9.9.9.9?
Comment 16 Dennis Kliban 2021-09-30 20:41:54 UTC 
While I was not able to reproduce this issue, I was able to reproduce the DNS errors that are present in the attached logs. There is a related bug in upstream aiohttp that was closed when it was determined that DNS throttling was causing the failures. This does not seem like a bug in Pulp or aiohttp. Please try to reproduce while DNS is configured to 9.9.9.9.

https://github.com/aio-libs/aiohttp/issues/5239#issuecomment-727544671
Comment 21 pulp-infra@redhat.com 2021-10-05 18:09:52 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 22 pulp-infra@redhat.com 2021-10-05 19:06:49 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 23 pulp-infra@redhat.com 2021-10-05 19:06:51 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 24 pulp-infra@redhat.com 2021-10-06 11:06:56 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 25 pulp-infra@redhat.com 2021-10-06 13:25:24 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 26 pulp-infra@redhat.com 2021-10-06 14:07:46 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 27 pulp-infra@redhat.com 2021-10-06 14:07:47 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 28 Lai 2021-10-11 14:24:50 UTC 
Steps to retest:

1. Install satellite 6.10 and import a manifest
2. Enable few small and big RHEL repos along with "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7"
3. Sync them
4. Try to resync "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7"

Expected:
Resyncing "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" should be successfull

Actual:
Resyncing "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" is successfull


Verified on 6.10_022
Comment 32 errata-xmlrpc 2021-11-16 14:13:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702
Note You need to log in before you can comment on or make changes to this bug.