Bug 2216453

Description of problem:
Having two Satellites configured as ISS via Network Sync (and overcoming TLSv1 issue with https://bugzilla.redhat.com/show_bug.cgi?id=2216445), an attempt to enable a Red Hat repository gets stuck.

Whenever I unwrap (on downstream Sat) a repository set (say, "Red Hat Enterprise Linux 7 Server - Optional (RPMs)" to see "x86_64 7.7" or "x86_64 7Server" or similar), Spinning button appears for a long time with "No repositories available." provided at the end. Despite upstream Satellite has a few repos enabled.

The ScanCDN task on the background is running infinitely, in a tight loop. Adding some debugs, it is recursivelly requesting more and more paths inside

https://github.com/katello/katello/blob/master/app/lib/katello/util/cdn_var_substitutor.rb#L75

that recursivelly calls itself for a growing list of "paths_with_substitutions", like:

2023-06-21T12:19:57 [I|app|] PavelM: find_substitutions: paths=[#<Katello::Util::PathWithSubstitutions:0x000055b9975b2b00 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], to_resolve=[#<Katello::Util::PathWithSubstitutions:0x000055b9975b2b00 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], resolved=[]
2023-06-21T12:19:58 [I|app|] PavelM: find_substitutions: paths=[#<Katello::Util::PathWithSubstitutions:0x000055b9973f6668 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f64b0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f62d0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f6118 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], to_resolve=[#<Katello::Util::PathWithSubstitutions:0x000055b9973f6668 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f64b0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f62d0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b9973f6118 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], resolved=[]
2023-06-21T12:19:59 [I|app|] PavelM: find_substitutions: paths=[#<Katello::Util::PathWithSubstitutions:0x000055b996ef1938 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef17f8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef1578 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef12f8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea3490 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea32b0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea3170 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea2fb8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed74c0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed7330 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed7150 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed6f70 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56c58 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56aa0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56910 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56668 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], to_resolve=[#<Katello::Util::PathWithSubstitutions:0x000055b996ef1938 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef17f8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef1578 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ef12f8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea3490 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea32b0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea3170 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ea2fb8 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed74c0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed7330 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed7150 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996ed6f70 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56c58 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56aa0 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56910 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">, #<Katello::Util::PathWithSubstitutions:0x000055b996e56668 @substitutions={}, @path="/content/dist/rhel/server/7/$releasever/$basearch/optional/os", @base_path="/content/dist/rhel/server/7/", @token="releasever">], resolved=[]




Version-Release number of selected component (if applicable):
Sat6.12 and Sat6.13 used; *both* directions for ISS failed the same way; I dont have two 6.13s or two 6.12s to test against same Sat version, but afaik this ISS feature isnt tighten to the same Sat Y version, no..?


How reproducible:
100% in my case


Steps to Reproduce:
0. Upload manifests for each Satellite (I used *different* manifests, maybe that is the cause..?)
1. On downstream Satellite, set ISS from an upstream Satellite 6: Content -> Subscriptions -> Manage Manifest -> CDN Configuration -> Network Sync -> provide upstream Sat details (incl. SSL debug cert)
2. Enable and Sync a RedHat repo on upstream Sat
3. On the downstream Sat, try to enable a Red Hat repo which is enabled on the upstream Sat
4. monitor /var/log/foreman/production.log on downstream Sat and also /var/log/httpd/foreman-ssl_access-ssl.log on upstream Sat.
5. Check running tasks on downstream Sat


Actual results:
3. long spinning with "No repositories available." at the end
4. shows TONS and TONS of queries like:

2023-06-21T14:57:31 [I|app|8681b97b] CDN: Requesting path https://pmoravec-sat613.gsslab.brq2.redhat.com:443/katello/api/v2/organizations/1/environments?full_result=true
2023-06-21T14:57:31 [I|app|] CDN: Requesting path https://pmoravec-sat613.gsslab.brq2.redhat.com:443/katello/api/v2/organizations/1/content_views?search=label%3DDefault_Organization_View

in production.log (downstream Sat) and

10.37.195.103 - - [21/Jun/2023:15:05:40 +0200] "GET /katello/api/v2/organizations/1/environments?full_result=true HTTP/1.1" 200 6082 "-" "Ruby"
10.37.195.103 - - [21/Jun/2023:15:05:40 +0200] "GET /katello/api/v2/repositories?full_result=true&organization_id=1&content_view_id=1&environment_id=1&search=content_label+%3D+rhel-7-server-optional-rpms HTTP/1.1" 200 4854 "-" "Ruby"

logs (and similar) in httpd access log (upstream Sat).

5. "Scan CDN" task running forever. Even forcefully stopping it does *not* stop new logs from 4.


Expected results:
repos to be shown, no end-less task or spammed logs


Additional info: