Bug 2148248 - Different boot.iso download depending on accepted encoding [NEEDINFO]
Summary: Different boot.iso download depending on accepted encoding
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: distribution
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Adam Samalik
QA Contact: Release Test Team
Depends On:
TreeView+ depends on / blocked
Reported: 2022-11-24 17:43 UTC by Gerald Vogt
Modified: 2022-12-05 14:10 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Type: Bug
Target Upstream Version:
asamalik: needinfo? (bstinson)

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-140509 0 None None None 2022-11-25 16:16:36 UTC

Description Gerald Vogt 2022-11-24 17:43:01 UTC
Description of problem:
When mirroring centos stream 9 using foreman/katello/pulp I have noticed frequent .treeinfo checksum problems with images/boot.iso (and maybe more but the sync aborts there).

Now I have noticed I get different files depending on whether I set http header accept-encoding to gzip or not.

How reproducible:

For example, downloading from http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso redirects me to https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso. One of the IPs of download.cf.centos.org is I do two downloads:

Steps to Reproduce:
1. curl -v --output boot1.iso --resolve download.cf.centos.org:443:  https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso --header 'Accept-Encoding: gzip'

2. curl -v --output boot2.iso --resolve download.cf.centos.org:443:  https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/images/boot.iso

3. ls -la boot?.iso                                                                                                                                        

4. file boot?.iso

5. shasum -a 256 boot?.iso

Actual results:
$ ls -la boot?.iso                                                                                                                                        
-rw-r--r--  1 root  root  896532480 Nov 24 17:22 boot1.iso
-rw-r--r--  1 root  root  893386752 Nov 24 17:22 boot2.iso

boot1.iso: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'CentOS-Stream-9-BaseOS-x86_64' (bootable)
boot2.iso: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'CentOS-Stream-9-BaseOS-x86_64' (bootable)

55a560c8fc43e3f2bfae3827a6e7c31fd45c1a72b61a2e9fc68f917cfddee9d1  boot1.iso
837ad493da7d544943066b040cd2ea87c37090307e96ce0f064687beb6821c33  boot2.iso

boot1.iso is the download with gzip which also matches the checksum in .treeinfo.

I have tried various IP addresses for download.cf.centos.org DNS, IPv4 and IPv6 addresses, it's always the same. So it's not only one particular mirror/server.

Expected results:

Both files should be identical and current.

Additional info:

I can see in the http response headers that the second is older then the first:

HTTP/2 200 
content-type: application/octet-stream
content-length: 896532480
date: Thu, 24 Nov 2022 01:00:30 GMT
last-modified: Wed, 23 Nov 2022 13:27:09 GMT
etag: "55c9647dc6665a4486d0b2d9569a48e0-107"
x-amz-version-id: S26qMpy9o2y1pZBxGs7Q7PB0esllhPYu
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 af1bbc213b3a9ee2f125be77ca3609a0.cloudfront.net (CloudFront)
x-amz-cf-pop: MUC50-P1
x-amz-cf-id: JvoKIs6ingsYWNwi3O4W705gph81_1gtEjEnKcsdnloI1Uub3JRh_Q==
age: 55305

this is the second:

HTTP/2 200 
content-type: application/octet-stream
content-length: 893386752
date: Tue, 22 Nov 2022 06:15:13 GMT
last-modified: Fri, 18 Nov 2022 13:16:06 GMT
etag: "5e9e8621cfaba576a3be45305bbce733-107"
x-amz-version-id: q8o94togmJky1WLWL3.Zt2CVQxOHepr4
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 551f2461af0b3bf4faaad831ee6e5b1e.cloudfront.net (CloudFront)
x-amz-cf-pop: MUC50-P1
x-amz-cf-id: P-GlbuHWF--HOLl3tICyYQ_DGScScTpaQDPNq-YO0REY7McJ3SyWfQ==
age: 209239

Comment 1 Adam Samalik 2022-12-05 14:10:30 UTC
Brian: Could there be something wrong with our cloudfront configuration?

Note You need to log in before you can comment on or make changes to this bug.