Bug 2148248

Description of problem:
When mirroring centos stream 9 using foreman/katello/pulp I have noticed frequent .treeinfo checksum problems with images/boot.iso (and maybe more but the sync aborts there).

Now I have noticed I get different files depending on whether I set http header accept-encoding to gzip or not.

How reproducible:

For example, downloading from http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso redirects me to https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso. One of the IPs of download.cf.centos.org is 18.66.192.118. I do two downloads:

Steps to Reproduce:
1. curl -v --output boot1.iso --resolve download.cf.centos.org:443:18.66.192.118  https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso --header 'Accept-Encoding: gzip'

2. curl -v --output boot2.iso --resolve download.cf.centos.org:443:18.66.192.118  https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso

3. ls -la boot?.iso                                                                                                                                        

4. file boot?.iso

5. shasum -a 256 boot?.iso

Actual results:
$ ls -la boot?.iso                                                                                                                                        
-rw-r--r--  1 root  root  896532480 Nov 24 17:22 boot1.iso
-rw-r--r--  1 root  root  893386752 Nov 24 17:22 boot2.iso

boot1.iso: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'CentOS-Stream-9-BaseOS-x86_64' (bootable)
boot2.iso: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'CentOS-Stream-9-BaseOS-x86_64' (bootable)

55a560c8fc43e3f2bfae3827a6e7c31fd45c1a72b61a2e9fc68f917cfddee9d1  boot1.iso
837ad493da7d544943066b040cd2ea87c37090307e96ce0f064687beb6821c33  boot2.iso

boot1.iso is the download with gzip which also matches the checksum in .treeinfo.

I have tried various IP addresses for download.cf.centos.org DNS, IPv4 and IPv6 addresses, it's always the same. So it's not only one particular mirror/server.

Expected results:

Both files should be identical and current.

Additional info:

I can see in the http response headers that the second is older then the first:

HTTP/2 200 
content-type: application/octet-stream
content-length: 896532480
date: Thu, 24 Nov 2022 01:00:30 GMT
last-modified: Wed, 23 Nov 2022 13:27:09 GMT
etag: "55c9647dc6665a4486d0b2d9569a48e0-107"
x-amz-version-id: S26qMpy9o2y1pZBxGs7Q7PB0esllhPYu
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 af1bbc213b3a9ee2f125be77ca3609a0.cloudfront.net (CloudFront)
x-amz-cf-pop: MUC50-P1
x-amz-cf-id: JvoKIs6ingsYWNwi3O4W705gph81_1gtEjEnKcsdnloI1Uub3JRh_Q==
age: 55305

this is the second:

HTTP/2 200 
content-type: application/octet-stream
content-length: 893386752
date: Tue, 22 Nov 2022 06:15:13 GMT
last-modified: Fri, 18 Nov 2022 13:16:06 GMT
etag: "5e9e8621cfaba576a3be45305bbce733-107"
x-amz-version-id: q8o94togmJky1WLWL3.Zt2CVQxOHepr4
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 551f2461af0b3bf4faaad831ee6e5b1e.cloudfront.net (CloudFront)
x-amz-cf-pop: MUC50-P1
x-amz-cf-id: P-GlbuHWF--HOLl3tICyYQ_DGScScTpaQDPNq-YO0REY7McJ3SyWfQ==
age: 209239