Bug 2104498

Created attachment 1894937 [details]
sync logs

This appears to be an issue in the aiohttp library that we use, I can reproduce it with aiohttp alone.

In [2]: import aiohttp
   ...: import asyncio
   ...: 
   ...: async def main():
   ...: 
   ...:     async with aiohttp.ClientSession() as session:
   ...:         async with session.get('https://releases.jfrog.io/artifactory/artifactory-pro-rpms/repodata/8c87521e43dbed223c90e23922c0c4bbe7159112-primary.xml.gz') as response:
   ...: 
   ...:             print("Status:", response.status)
   ...:             print("Content-type:", response.headers['content-type'])
   ...: 
   ...:             html = await response.text()
   ...:             print("Body:", html)
   ...: 
   ...: loop = asyncio.get_event_loop()
   ...: loop.run_until_complete(main())
Status: 403
Content-type: text/xml
Body: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access denied</Message></Error>


Artifactory serves up a redirect which aiohttp doesn't seem to encode to the webserver's liking.  The top URL is the redirect URL that pulp tries to use, the bottom one is the one that wget and httpie use.  If you look at the "X-Artifactory-artifactPath" parameter, the one on the top uses a forward slash and the one on bottom uses urlencoding.

https://releases-cdn.jfrog.io/filestore/8c/8c87521e43dbed223c90e23922c0c4bbe7159112?response-content-type=application/x-gzip&response-content-disposition=attachment%3Bfilename%3D%228c87521e43dbed223c90e23922c0c4bbe7159112-primary.xml.gz%22&x-jf-traceId=da959416009abe79&X-Artifactory-repositoryKey=artifactory-pro-rpms&X-Artifactory-projectKey=default&X-Artifactory-artifactPath=repodata/8c87521e43dbed223c90e23922c0c4bbe7159112-primary.xml.gz&X-Artifactory-username=anonymous&Expires=1657248968&Signature=MA21YnR3A8BncqlPFTcELv15ndn2B6yNQpaag2JuuPiWgiwjYnGHlbQkDbXp6Gk3ygrKOPvUETeb-gSzv6g64kFVnNzYzDxMRdV71nIP2YRWkufG-2R9AGSA9OtAevmAWYG-wmGazZt0L6VqX-4XDfpQPoVG5RITCMzQnQ3W~XbX8lB2AhliU0GI8QNOFzVKB8bAiFIFjh0HBV-P~~DlRaOn0ouKkdS-6paLjDq7NPEmlXrt2A~oOd5NvwViRpUNSsbov6WugJHPMMxPb5wj3B7hISYfwN6~6TWgbZ8Y6o3GXV-3zhxN4idpkFuvF90d2Aoep~gmeGKUysk3EohzVQ__&Key-Pair-Id=APKAJ6NHFWMVU3M6DPBA

https://releases-cdn.jfrog.io/filestore/8c/8c87521e43dbed223c90e23922c0c4bbe7159112?response-content-type=application/x-gzip&response-content-disposition=attachment%3Bfilename%3D%228c87521e43dbed223c90e23922c0c4bbe7159112-primary.xml.gz%22&x-jf-traceId=e4edec532ab037c3&X-Artifactory-repositoryKey=artifactory-pro-rpms&X-Artifactory-projectKey=default&X-Artifactory-artifactPath=repodata%2F8c87521e43dbed223c90e23922c0c4bbe7159112-primary.xml.gz&X-Artifactory-username=anonymous&Expires=1657249015&Signature=dwsfXduAz0Avg62JQfIgXuQGP2cLv6AExW4u9gR3B0wzOu5R599u~satGkD6yLIzB4Y1uBCLhU6IO7b-ovr0wT9AJTxMaE-eiF2awcaj2eGzXpFibJVoUuzhsG3CW5koby8cytTThxRYewn1nACLna58og14o~EkXIgA3~c32y5ltR1rtJDyclQ~wpQNTpc4vX4uk2hr5z0lolmCGy0fgLxmuiFkzNmsHFAUbo1qqb8kJF~SAE-l4erLPAIlyIuDpAl6myyJ1cHmS-dolXiz8M6yNel-sJY5jaOH-iAlZoZm9PHdqmKbMqrzdgzLS2emZF5zER9g068lY~7fLqgqrw__&Key-Pair-Id=APKAJ6NHFWMVU3M6DPBA

Root cause: https://github.com/aio-libs/yarl/issues/245

Although going by some of the comments, it's plausible that Yarl is abiding strictly by the spec, whereas whichever webserver Artifactory is using is rejecting requests that it ought to accept, because '/' is (supposed to be?) an allowed character.

It seems like most other implementations are more aggressive with urlencoding than Yarl, however.  As mentioned there's wget and httpie, but I believe also urllib.

A decent argument can also be made for https://github.com/aio-libs/aiohttp/issues/5319 being the "real" issue.

@dalley try https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.requote_redirect_url to see if it helps.

Yes, this works.  Patch incoming.

*** Bug 2115882 has been marked as a duplicate of this bug. ***

*** Bug 2115878 has been marked as a duplicate of this bug. ***

Verified in 6.12 snap 6

Repository sync runs successfully with URL: https://releases.jfrog.io/artifactory/artifactory-pro-rpms/

Steps to Reproduce:

1) Create a new product and yum repo with URL set to https://releases.jfrog.io/artifactory/artifactory-pro-rpms/

2) Sync the repository

Expected Results:

Sync runs successfully and all rpms are downloaded.

Actual Results:

Sync runs successfully and all rpms are downloaded.


Notes:

This repo is quite large and requires over 100Gs to sync. Took my Satellite about 30 minutes to finish.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506