Description of problem: When download the RHCOS image from the mirror.openshift.com, the specified size of the image is 656M, but in fact is 1.8G. When using a browser like Chrome to download the image, we get the full image. When doing "right click and Copy link address" and using this address with "wget" or "curl", the image is corrupt and has only 656M . Version-Release number of selected component (if applicable): How reproducible: 1 - Use a browser to goto: https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/ 2 - Right click on rhcos-42.80.20190828..> and Click on "Copy link address" 3 - Download the image using wget [link from clipboard] 4 - The image size will be 656M . 5 - Do qemu-img info. The image format will be RAW and will not boot. Actual results: [root@osp14ocp42 ~(keystone_ocp)]# qemu-img info rhcos-42.80.20190828.2-openstack.qcow2 image: rhcos-42.80.20190828.2-openstack.qcow2 file format: raw virtual size: 656M (687595008 bytes) disk size: 656M Expected results: [root@osp14ocp42 mouimet(keystone_ocp)]# qemu-img info rhcos-42.80.20190828.2-openstack.qcow2 image: rhcos-42.80.20190828.2-openstack.qcow2 file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 1.8G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false Additional info:
The OpenStack qcow2 images are shipped in compressed format. We need to document this, so that users can correctly consume them. See the following header when requesting via `curl`: ``` $ curl -I https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-42.80.20190828.2-openstack.qcow2 HTTP/1.1 200 OK Date: Wed, 25 Sep 2019 17:49:54 GMT Server: Apache/2.2.15 (Red Hat) Last-Modified: Fri, 06 Sep 2019 13:18:09 GMT ETag: "422bcc14-28fbdc0f-591e244f38640" Accept-Ranges: bytes Content-Length: 687594511 Strict-Transport-Security: max-age=15768000, includeSubDomains Connection: close Content-Type: application/octet-stream Content-Encoding: x-gzip ``` Notice the `Content-Encoding` is `x-gzip`. Clients can be instructed to decompress the image on the fly; it's likely that your browser does that automatically, whereas `curl` and `wget` need explicit options provided. Users should use `curl --compressed -J -L -O <url>` which will correctly retrieve the image: ``` $ curl --compressed -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-42.80.20190828.2-openstack.qcow2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 655M 100 655M 0 0 924k 0 0:12:06 0:12:06 --:--:-- 969k $ file rhcos-42.80.20190828.2-openstack.qcow2 rhcos-42.80.20190828.2-openstack.qcow2: QEMU QCOW2 Image (v3), 17179869184 bytes $ ls -l rhcos-42.80.20190828.2-openstack.qcow2 -rw-rw-r--. 1 miabbott miabbott 1885470720 Sep 25 14:03 rhcos-42.80.20190828.2-openstack.qcow2 $ qemu-img info rhcos-42.80.20190828.2-openstack.qcow2 image: rhcos-42.80.20190828.2-openstack.qcow2 file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 1.8G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false ``` Or using `wget --compression=auto <url>`: ``` $ wget --compression=auto https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-42.80.20190828.2-openstack.qcow2 --2019-09-25 14:10:24-- https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-42.80.20190828.2-openstack.qcow2 Resolving mirror.openshift.com (mirror.openshift.com)... 54.173.18.88, 54.172.163.83, 54.172.173.155 Connecting to mirror.openshift.com (mirror.openshift.com)|54.173.18.88|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 687594511 (656M) [application/octet-stream] Saving to: ‘rhcos-42.80.20190828.2-openstack.qcow2’ rhcos-42.80.20190828.2-openstack.qcow2 100%[===============================================>] 655.74M 542KB/s in 16m 27s 2019-09-25 14:26:51 (680 KB/s) - ‘rhcos-42.80.20190828.2-openstack.qcow2’ saved [1885470720] $ file rhcos-42.80.20190828.2-openstack.qcow2 rhcos-42.80.20190828.2-openstack.qcow2: QEMU QCOW2 Image (v3), 17179869184 bytes $ ls -l rhcos-42.80.20190828.2-openstack.qcow2 -rw-rw-r--. 1 miabbott miabbott 1885470720 Sep 6 09:18 rhcos-42.80.20190828.2-openstack.qcow2 $ qemu-img info rhcos-42.80.20190828.2-openstack.qcow2 image: rhcos-42.80.20190828.2-openstack.qcow2 file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 1.8G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false ``` Lastly, users should use `gunzip` to decompress the image, if the appropriate flags to `curl` or `wget` are not provided: ``` $ curl -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-42.80.20190828.2-openstack.qcow2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 655M 100 655M 0 0 1026k 0 0:10:54 0:10:54 --:--:-- 2654k $ file rhcos-42.80.20190828.2-openstack.qcow2 rhcos-42.80.20190828.2-openstack.qcow2: gzip compressed data, was "rhcos-42.80.20190828.2-openstack.qcow2", last modified: Wed Aug 28 14:05:49 2019, from Unix, original size 1885470720 $ ls -l rhcos-42.80.20190828.2-openstack.qcow2 -rw-rw-r--. 1 miabbott miabbott 687594511 Sep 25 14:44 rhcos-42.80.20190828.2-openstack.qcow2 $ gunzip -S .qcow2 rhcos-42.80.20190828.2-openstack.qcow2 $ file rhcos-42.80.20190828.2-openstack rhcos-42.80.20190828.2-openstack: QEMU QCOW2 Image (v3), 17179869184 bytes $ ls -l rhcos-42.80.20190828.2-openstack -rw-rw-r--. 1 miabbott miabbott 1885470720 Sep 25 14:44 rhcos-42.80.20190828.2-openstack $ qemu-img info rhcos-42.80.20190828.2-openstack image: rhcos-42.80.20190828.2-openstack file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 1.8G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false ```
Need to update the documentation to consume compress images using WGET https://github.com/openshift/installer/tree/master/docs/user/openstack
PR to docs - https://github.com/openshift/openshift-docs/pull/16891
PR to installer user docs - https://github.com/openshift/installer/pull/2416