Description of problem: Error: " isoSize: Atoi error on endpoint "file/path/file_name.raw.xz": strconv.Atoi: parsing "f5": invalid syntax " appears in importer pod log, when importing a raw.xz file. Version-Release number of selected component (if applicable): virt-cdi-importer:v1.4.0 virt-cdi-cloner:v1.4.0 virt-cdi-uploadserver:v1.4.0 virt-cdi-controller:v1.4.0-7 How reproducible: 100% Steps to Reproduce: 1. Create a PVC: oc create -f pvc.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: "https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz" cdi.kubevirt.io/storage.import.secretName: "" cdi.kubevirt.io/storage.import.source: http name: pvc-on-gluster-for-vmi-fedora spec: accessModes: - ReadWriteOnce resources: requests: storage: 12Gi selector: ~ storageClassName: glusterfs-storage 2. View importer log: oc log -f importer_pod_name Actual results: The following error appeared: E0210 22:24:40.100489 1 dataStream.go:617] isoSize: Atoi error on endpoint "/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz": strconv.Atoi: parsing "f5": invalid syntax Expected results: No errors in importer pod log. Additional info: 1. This error seem to not have any impact. 2. When importing a raw file this error doesn't show. 3. Full log: $ oc logs -f importer-pvc-on-gluster-for-vmi-fedora-6nsvs I0210 22:24:39.597762 1 importer.go:45] Starting importer I0210 22:24:39.598048 1 importer.go:64] begin import process I0210 22:24:39.598054 1 importer.go:88] begin import process I0210 22:24:39.598079 1 dataStream.go:293] copying "https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz" to "/data/disk.img"... E0210 22:24:40.100489 1 dataStream.go:617] isoSize: Atoi error on endpoint "/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz": strconv.Atoi: parsing "f5": invalid syntax I0210 22:24:40.113342 1 util.go:38] begin import... I0210 22:27:10.308678 1 prlimit.go:107] ExecWithLimits qemu-img, [info --output=json /data/disk.img] W0210 22:27:11.322559 1 dataStream.go:343] Available space less than requested size, resizing image to available space 8415285248. I0210 22:27:11.322588 1 dataStream.go:349] Expanding image size to: 8415285248 I0210 22:27:11.323975 1 prlimit.go:107] ExecWithLimits qemu-img, [resize -f raw /data/disk.img 8415285248] I0210 22:27:14.154757 1 importer.go:95] import complete
Natalie does the qcow2 import work? https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2 If it does then I agree this is could be "medium" severity, if it doesn't (because of the same or other bugs) than this becomes more serious because Fedora cannot be imported at all (only raw.xz and qcow2 formats are available).
(In reply to Federico Simoncelli from comment #1) > Natalie does the qcow2 import work? > > https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/ > images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2 > > If it does then I agree this is could be "medium" severity, if it doesn't > (because of the same or other bugs) than this becomes more serious because > Fedora cannot be imported at all (only raw.xz and qcow2 formats are > available). Federico, My only complaint is the error, the import itself works. To make clear, both formats (qcow2 and raw.xz) work - the bug is about the error: "isoSize: Atoi error on endpoint" that is shown in the importer pod log when importing a raw.xz file.
(In reply to Natalie Gavrielov from comment #2) > (In reply to Federico Simoncelli from comment #1) > > Natalie does the qcow2 import work? > > > > https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/ > > images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2 > > > > If it does then I agree this is could be "medium" severity, if it doesn't > > (because of the same or other bugs) than this becomes more serious because > > Fedora cannot be imported at all (only raw.xz and qcow2 formats are > > available). > > Federico, > > My only complaint is the error, the import itself works. > To make clear, both formats (qcow2 and raw.xz) work - the bug is about the > error: "isoSize: Atoi error on endpoint" that is shown in the importer pod > log when importing a raw.xz file. So is it a text-only issue? Any other possible side-effect that we know of? (Question goes to both you Natalie and Adam)
I saw this too. It's really just an overly verbose message when detecting the file type. To fix we need only to suppress this from the logs. As such, medium severity is appropriate.
(In reply to Adam Litke from comment #4) > I saw this too. It's really just an overly verbose message when detecting > the file type. To fix we need only to suppress this from the logs. As > such, medium severity is appropriate. IIUC this double conversion seems fishy: vdtyp, err := strconv.Atoi(hex.EncodeToString(...)) why should we convert to string to re-convert to integer to check difference (vdtyp != primaryVD)? Instead of just suppressing the log I hope we can improve the detection logic. Should this be assigned to someone?
So a few interesting things are going on here. 1. The weird conversion is basically 2 bytes that are a hex representation of a number, that is why the hex encode and then conversion to int happens. 2. That particular input file is a raw file, but not a standard raw file. Normal raw files have a volume descriptor at a particular offset in the file that we use to determine if the file is raw (well ISO9660 format to be exact). That one has the volume descriptor but its at the end of the file for some reason. A quick search for different raw/iso formats didn't surface any obvious different formats to look at. 3. The conversion from hex string back to int should be done with strconv.ParseInt(string, 16, 32) instead of Atoi which doesn't accept the value that was in the two bytes of the image in the location we looked (it happens to be F5) So I have fixed the conversion to be correct in case the value is not 1 (which is why the atoi works in normal cases). We also should investigate more these raw formats that do not comply with standard ISO9660 formats but are still raw, and we should support writing them directly to the target instead of needed intermediate scratch space.
Created issue in GH issues to track the strange RAW format in the file.
Verified, using scenario and yaml from comment 0. importer pod output: # oc logs -f importer-pvc-for-vmi-fedora-pqnvs I0512 14:00:22.400827 1 importer.go:58] Starting importer I0512 14:00:22.401099 1 importer.go:100] begin import process I0512 14:00:24.199292 1 data-processor.go:230] Calculating available size I0512 14:00:24.206318 1 data-processor.go:238] Checking out file system volume size. I0512 14:00:24.206352 1 data-processor.go:242] Request image size not empty. I0512 14:00:24.206363 1 data-processor.go:247] Target size 12Gi. I0512 14:00:24.295571 1 data-processor.go:160] New phase: TransferDataFile I0512 14:00:24.301123 1 util.go:43] Writing data... I0512 14:02:37.288323 1 data-processor.go:166] Validating image I0512 14:02:37.312091 1 data-processor.go:160] New phase: Resize I0512 14:02:37.346499 1 data-processor.go:223] Expanding image size to: 12Gi I0512 14:02:37.418945 1 data-processor.go:160] New phase: Complete I0512 14:02:37.418968 1 importer.go:132] import complete Build: 2.0.0-5 (CDI 1.9)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:1850