Bug 1993773
| Summary: | Synchronization of "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" fails with an unhelpful error "Pulp task error" | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Sayan Das <saydas> |
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED ERRATA | QA Contact: | Lai <ltran> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.10.0 | CC: | ahumbe, dalley, dkliban, ggainey, gpayelka, jsherril, pmendezh, rchan, ttereshc, zhunting |
| Target Milestone: | 6.10.0 | Keywords: | Triaged |
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-16 14:13:06 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I have checked it again on a 4vCPU\20 GB ram satellite 6.10, The issue remains same. If I try to sync EUS 7.7 with any other big repos at once, all repos will get sync except EUS 7.7 and the point of failure is exactly the same what was reported in the Description. *** Bug 1996106 has been marked as a duplicate of this bug. *** The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug. The Pulp upstream bug priority is at High. Updating the external tracker on this bug. The Pulp upstream bug status is at NEW. Updating the external tracker on this bug. The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug. It's possible that this bug is related to a rate limiting caused by the DNS servers. Could you please try to reproduce when the DNS is configured to use 9.9.9.9? While I was not able to reproduce this issue, I was able to reproduce the DNS errors that are present in the attached logs. There is a related bug in upstream aiohttp that was closed when it was determined that DNS throttling was causing the failures. This does not seem like a bug in Pulp or aiohttp. Please try to reproduce while DNS is configured to 9.9.9.9. https://github.com/aio-libs/aiohttp/issues/5239#issuecomment-727544671 The Pulp upstream bug status is at POST. Updating the external tracker on this bug. The Pulp upstream bug status is at NEW. Updating the external tracker on this bug. The Pulp upstream bug priority is at High. Updating the external tracker on this bug. The Pulp upstream bug status is at POST. Updating the external tracker on this bug. The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST. Steps to retest: 1. Install satellite 6.10 and import a manifest 2. Enable few small and big RHEL repos along with "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" 3. Sync them 4. Try to resync "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" Expected: Resyncing "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" should be successfull Actual: Resyncing "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" is successfull Verified on 6.10_022 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702 |
Description of problem: Attempted to sync RHEL 7.7 EUS repo twice and both time it failed with some very unhelpful error i.e. "Pulp task error". Version-Release number of selected component (if applicable): - Satellite 6.10.0 [ latest snap satellite-6.10.0-0.6.beta.el7sat.noarch ] How reproducible: - Always [ At least two consecutive attempts failed ] Steps to Reproduce: 1. Install satellite 6.10 and import a manifest 2. Enable few small and big RHEL repos along with "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" 3. Sync them 4. Try to resync "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" Actual results: Sync of "Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7" will always fail with following error in UI. "Pulp task error" And folloing traceback in syslog : Aug 16 10:57:52 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: - - [16/Aug/2021:05:27:52 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 1035 "-" "OpenAPI-Generator/3.14.1/ruby" Aug 16 10:58:08 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: - - [16/Aug/2021:05:28:08 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 1035 "-" "OpenAPI-Generator/3.14.1/ruby" Aug 16 10:58:22 satellite pulpcore-worker-1: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: pulpcore.tasking.pulpcore_worker:INFO: Task 7bb636dc-3b2a-4f7f-99b2-56f6c0164405 failed () Aug 16 10:58:22 satellite pulpcore-worker-1: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task Aug 16 10:58:22 satellite pulpcore-worker-1: result = func(*args, **kwargs) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 427, in synchronize Aug 16 10:58:22 satellite pulpcore-worker-1: version = dv.create() Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create Aug 16 10:58:22 satellite pulpcore-worker-1: loop.run_until_complete(pipeline) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete Aug 16 10:58:22 satellite pulpcore-worker-1: return future.result() Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline Aug 16 10:58:22 satellite pulpcore-worker-1: await asyncio.gather(*futures) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__ Aug 16 10:58:22 satellite pulpcore-worker-1: await self.run() Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 659, in run Aug 16 10:58:22 satellite pulpcore-worker-1: name, location_href, result = await future Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/asyncio/tasks.py", line 458, in _wait_for_one Aug 16 10:58:22 satellite pulpcore-worker-1: return f.result() # May raise f.exception(). Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 631, in run_repomdrecord_download Aug 16 10:58:22 satellite pulpcore-worker-1: result = await downloader.run() Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run Aug 16 10:58:22 satellite pulpcore-worker-1: return await download_wrapper() Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry Aug 16 10:58:22 satellite pulpcore-worker-1: ret = await target(*args, **kwargs) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper Aug 16 10:58:22 satellite pulpcore-worker-1: return await self._run(extra_data=extra_data) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 95, in _run Aug 16 10:58:22 satellite pulpcore-worker-1: to_return = await self._handle_response(response) Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 210, in _handle_response Aug 16 10:58:22 satellite pulpcore-worker-1: chunk = await response.content.read(1048576) # 1 megabyte Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read Aug 16 10:58:22 satellite pulpcore-worker-1: await self._wait("read") Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait Aug 16 10:58:22 satellite pulpcore-worker-1: await waiter Aug 16 10:58:22 satellite pulpcore-worker-1: File "/usr/lib64/python3.6/site-packages/aiohttp/helpers.py", line 656, in __exit__ Aug 16 10:58:22 satellite pulpcore-worker-1: raise asyncio.TimeoutError from None Aug 16 10:58:24 satellite pulpcore-api: pulp [c7197c3c-5399-45d6-8f14-ad242389d20d]: - - [16/Aug/2021:05:28:24 +0000] "GET /pulp/api/v3/tasks/7bb636dc-3b2a-4f7f-99b2-56f6c0164405/ HTTP/1.1" 200 3393 "-" "OpenAPI-Generator/3.14.1/ruby" Expected results: 1. The repo should be synced properly. 2. If the repo sync is failing, an understandable Error message or traceback should be printed in foreman-tasks or syslogs so that we can come to know what is the exact issue and why it failed. Additional info: - Task export for both of the failed sync attempts and /var/log/messages from satellite server will be uploaded on the case. - It could be related to a low memory situation but the error is not clear at all and pulp2 can sync this repo without any issues. - NOTE: All services in satellite is running fine and "hammer ping" shows OK for all components.