Bug 1462189 - ReaR cannot create ISO images larger than 4GB due to Red Hat change that removes mkisofs
ReaR cannot create ISO images larger than 4GB due to Red Hat change that remo...
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: cdrkit (Show other bugs)
7.3
All Linux
unspecified Severity urgent
: rc
: ---
Assigned To: Vaclav Dolezal
BaseOS QE - Apps
: Reopened
Depends On:
Blocks: 1473612
  Show dependency treegraph
 
Reported: 2017-06-16 08:04 EDT by afox@redhat.com
Modified: 2017-08-17 10:36 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-29 15:25:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description afox@redhat.com 2017-06-16 08:04:22 EDT
Description of problem:
The version of ReaR that is shipped with RHEL7 does not include the mkisofs binary. Instead, we substitute genisoimage. 

As a result of this change, ISO images greater than 4GB cannot be created. 

See:
https://github.com/rear/rear/issues/581
for documentation of this issue on the ReaR github issue tracker.

The customer has attempted to use the workaround documented in the issue tracker. This does result in the creation of an ISO image greater than 4GB. 

However, when attempting to restore using this image, the following error is seen:

Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-lonrs07335.log
ERROR: Mount command 'mount /dev/disk/by-label/RELAXRECOVER /tmp/rear.KPTy08rH3CYH3Bq/outputfs' failed.
Aborting due to an error, check /var/log/rear/rear-lonrs07335.log for details
/etc/scripts/system-setup: line 72: 4753 Terminated     rear recover -v


Version-Release number of selected component (if applicable):
Relax-and-Recover 1.17.2

How reproducible:
Always. 

Steps to Reproduce:
1. Attempt to image a system where ISO size will be greater than 4GB. 


Actual results:
ISO image is not created. 

Expected results:
Working ISO image should be created. 

Additional info:
This represents a major problem for RBS as ReaR is at the core of all their disaster recovery. Their latest shipping RHEL build results in an ISO image size of greater than 4GB, so currently they cannot ship this build until ReaR is fixed.
Comment 2 Jakub Mazanek 2017-06-29 15:25:14 EDT
This is not a rear issue. It is a behaviour of genisoimage package. Nothing I can do.

Jakub
Comment 3 Julio Entrena Perez 2017-07-03 05:34:02 EDT
Re-assigning to cdrkit (genisoimage) as per comment 2.
Comment 4 Vaclav Dolezal 2017-07-18 07:24:04 EDT
genisoimage has switch -allow-limited-size which creates hybrid UDF/ISO9660 filesystem where size info is correct only on UDF metadata and big files are truncated on ISO9660 metadata. Resulting filesystem has to be mounted as UDF but it can represent files greater than 4GiB.

Can it be used by ReaR?
Comment 5 Julio Entrena Perez 2017-07-18 10:01:20 EDT
# rear -v mkbackup
Relax-and-Recover 1.17.2 / Git
Using log file: /var/log/rear/rear-rhel7a34.log
Creating disk layout
Creating root filesystem layout
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Encrypting disabled
Creating tar archive '/tmp/rear.zh4FTegowf9b0il/tmp/isofs/backup//backup.tar.gz'
Archived 4753 MiB [avg 5921 KiB/sec]OK
Archived 4753 MiB in 823 seconds [avg 5914 KiB/sec]
Making ISO image
ERROR: Could not create ISO image (with /bin/mkisofs)
Aborting due to an error, check /var/log/rear/rear-rhel7a34.log for details
Terminated

# cat /var/log/rear/rear-rhel7a34.log
2017-07-18 13:40:12 Relax-and-Recover 1.17.2 / Git
2017-07-18 13:40:12 Command line options: /usr/sbin/rear -v mkbackup
[...]
2017-07-18 13:55:50 Including output/ISO/Linux-i386/81_prepare_multiple_iso.sh
2017-07-18 13:55:50 Including output/ISO/Linux-i386/82_create_iso_image.sh
2017-07-18 13:55:50 Starting '/bin/mkisofs'
2017-07-18 13:55:50 Making ISO image
genisoimage 1.1.11 (Linux)
Scanning .
Scanning ./backup
File ./backup/backup.tar.gz is larger than 4GiB-1.
-allow-limited-size was not specified. There is no way do represent this file size. Aborting.
2017-07-18 13:55:50 ERROR: Could not create ISO image (with /bin/mkisofs)
=== Stack trace ===
Trace 0: /usr/sbin/rear:252 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:22 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:70 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:31 Source
Trace 4: /usr/share/rear/output/ISO/Linux-i386/82_create_iso_image.sh:17 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:132 StopIfError
Message: Could not create ISO image (with /bin/mkisofs)

# man genisoimage
[...]
       -allow-limited-size
              When processing files larger than 2GiB which cannot be easily represented in ISO9660, add them with a shrunk visible file size  to  ISO9660
              and  with  the  correct visible file size to the UDF system. The result is an inconsistent filesystem and users need to make sure that they
              really use UDF rather than ISO9660 driver to read a such disk. Implies enabling -udf.

(In reply to Vaclav Dolezal from comment #4)
> Resulting filesystem has to be mounted as UDF
> but it can represent files greater than 4GiB.
> 
> Can it be used by ReaR?
Comment 7 Frantisek Kluknavsky 2017-07-28 04:13:16 EDT
Hi,

please, how exactly is mkisofs invoked by rear? (Is it written somewhere in that /var/log/rear/rear-rhel7a34.log?)
Comment 8 Julio Entrena Perez 2017-07-28 05:14:02 EDT
/usr/share/rear/output/ISO/Linux-i386/82_create_iso_image.sh :

     13 $ISO_MKISOFS_BIN $v -o "$ISO_DIR/$ISO_PREFIX.iso" -b isolinux/isolinux.bin -c isolinux/boot.cat \
     14     -no-emul-boot -boot-load-size 4 -boot-info-table \
     15     -R -J -volid "$ISO_VOLID" $EFIBOOT -v -iso-level 3 .  >&8
     16     ##-R -J -volid "$ISO_VOLID" $EFIBOOT  "${ISO_FILES[@]}"  >&8

Btw, in RHEl mkisofs is actually genisoimage:

# which mkisofs
/usr/bin/mkisofs

# ls -l /usr/bin/mkisofs 
lrwxrwxrwx. 1 root root 25 Jul 18 12:51 /usr/bin/mkisofs -> /etc/alternatives/mkisofs

# ls -l /etc/alternatives/mkisofs
lrwxrwxrwx. 1 root root 20 Jul 18 12:51 /etc/alternatives/mkisofs -> /usr/bin/genisoimage
Comment 10 Frantisek Kluknavsky 2017-07-29 16:05:18 EDT
Trying to include a large file into the initrd results in a small iso that does not contain that file.
900_create_initramfs.sh contains:

find . ! -name "*~"  |\
        tee /dev/fd/8  |\
        cpio -H newc --create --quiet  |\
        gzip > "$TMP_DIR/initrd.cgz"

That does not check for errors much. It leaves an error message in the log and continues with an incomplete archive:
"cpio: usr/share/rear/dummy.bin: field width not sufficient for storing file size"
Libarchive man page says: "SVR4/newc ... The SVR4 format uses eight-digit hexadecimal values for all header fields. This limits file size to 4GB, and also limits the mtime and other fields to 32 bits."

As Vaclav says, -allow-limited-size creates an udf filesystem. I was not able to boot it. Booting it as iso9660 leads to kernel panic. Please give me a hint if you can boot from it properly as udf. Reading it as udf on a working machine works well.

Xorrisofs (insteat of genisoimage) seems to create a correct 6GB large iso9660 image. But I am not able to boot it either. Virtual machine with 16GB of ram reports "not enough memory" in the middle of loading initrd. Again, please give me a hint if you know possible reasons.
Comment 11 Julio Entrena Perez 2017-07-31 07:16:13 EDT
(In reply to Frantisek Kluknavsky from comment #10)
 
> Xorrisofs (insteat of genisoimage) seems to create a correct 6GB large
> iso9660 image. But I am not able to boot it either. 

Thanks for the suggestion on Xorrisofs, I can confirm that with rear-2.00-2.el7 (included in RHEL 7.4) and xorriso 1.3.6 I can successfully produce a single, larger than 4GB ISO image:

# rear -v mkbackup
Relax-and-Recover 2.00 / Git
Using log file: /var/log/rear/rear-rhel7a34.log
Using backup archive '/tmp/rear.GG8Nc1BZhpqCqhZ/tmp/isofs/backup/backup.tar.gz'
Creating disk layout
Creating root filesystem layout
Copying logfile /var/log/rear/rear-rhel7a34.log into initramfs as '/tmp/rear-rhel7a34-partial-2017-07-31T10:07:34+0000.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Creating tar archive '/tmp/rear.GG8Nc1BZhpqCqhZ/tmp/isofs/backup/backup.tar.gz'
Archived 5715 MiB [avg 5600 KiB/sec] OK
Archived 5715 MiB in 1046 seconds [avg 5594 KiB/sec]
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-rhel7a34.iso (5.8G)
Copying resulting files to file location
Saving /var/log/rear/rear-rhel7a34.log as rear-rhel7a34.log to file location

# ls -lh /var/lib/rear/output
total 5.8G
-rw-------. 1 root root 5.8G Jul 31 10:40 rear-rhel7a34.iso


I can also confirm that I can successfully boot and restore from the produced ISO image.

Note You need to log in before you can comment on or make changes to this bug.