A shorter reproducer is to pick up the qcow2 image used by the installer from ~/.cache/openshift-install/libvirt/image/ , unzip it, and then run: $ TMPDIR=$(pwd) virt-sparsify ./rhcos-43.81.201911081536.0-qemu.x86_64.qcow2 sparse.qcow2
The filesystem on /dev/sda4 is XFS and mounts fine (there are no kernel messages indicating any problem) but I simply cannot write to it even as root. All writes return -EPERM but there's no other indication of what is wrong.
$ virt-rescue --ro -a rhcos-43.81.201911081536.0-qemu.x86_64.qcow2 ><rescue> file -bsL /dev/sda4 SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs) ><rescue> mount /dev/sda4 /sysroot/ ><rescue> dmesg | tail [ 0.413665] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.413709] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.414962] sd 2:0:1:0: [sdb] Attached SCSI disk [ 0.415296] sda: sda1 sda2 sda3 sda4 [ 0.415732] sd 2:0:0:0: [sda] Attached SCSI disk [ 0.421296] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem [ 0.423023] EXT4-fs (sdb): mounted filesystem without journal. Opts: [ 22.706196] SGI XFS with ACLs, security attributes, scrub, no debug enabled [ 22.709669] XFS (sda4): Mounting V5 Filesystem [ 22.715660] XFS (sda4): Ending clean mount ><rescue> ls -la /sysroot total 8 drwxr-xr-x. 4 root root 66 Nov 8 15:43 . drwxr-xr-x 19 root root 4096 Nov 12 16:02 .. -rw-rw-r--. 1 root root 257 Nov 8 15:43 .coreos-aleph-version.json drwxrwxr-x. 2 root root 6 Nov 8 15:42 boot drwxrwxr-x. 5 root root 62 Nov 8 15:43 ostree ><rescue> touch /sysroot/hello touch: setting times of '/sysroot/hello': No such file or directory ><rescue> strace touch /sysroot/hello execve("/bin/touch", ["touch", "/sysroot/hello"], 0x7ffe6aacbb68 /* 13 vars */) = 0 [...] openat(AT_FDCWD, "/sysroot/hello", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 EPERM (Operation not permitted) utimensat(AT_FDCWD, "/sysroot/hello", NULL, 0) = -1 ENOENT (No such file or directory)
It turns out (thanks Jeff Moyer) that the root directory of that filesystem is marked as immutable: ><rescue> lsattr -a /sysroot ----i--------------- /sysroot/. which is what causes the problem. Because of the way that virt-sparsify works in copying mode it needs to temporarily write a balloon file to each filesystem, so an immutable filesystem isn't going to work. As a workaround for this particular disk you can use the --ignore option, eg: virt-sparsify --ignore /dev/sda4 [...] You can also use in-place sparsification instead, which is much faster. This ignores the immutable bit and I've checked that it works on this particular filesystem.
@Richard Thank you for quick look and providing a workaround. Before we were using `in-place` option but that doesn't work when we specify `-o lazy_refcounts=on` option and that is something we really required to make this qcow2 image work with mac using hyperkit. We will try to use the --ignore option meanwhile and see if resulted image is work for our case. I still not sure why this change is required for Redhat CoreOS side, may be @Micah or @Steve can answer.
Another thing we were discussing is temporarily removing the immutable attribute around the virt-sparsify command, ie: $ guestfish -a source.qcow2 -m /dev/sda4 set-e2attrs / i clear:true $ virt-sparsify source.qcow2 dest.qcow2 $ guestfish -a dest.qcow2 -m /dev/sda4 set-e2attrs / i Although the command is called set-e2attrs it can in fact set XFS attributes too, but see also bug 1771587.
Also when I inspect the Redhat CoreOS image for 4.2 and 4.3 side I see the difference in mount points. - 4.3 images ``` $ guestfish <<_EOF_ add crc.qcow2 run list-filesystems _EOF_ /dev/sda1: ext4 /dev/sda2: vfat /dev/sda3: unknown /dev/sda4: xfs ``` - 4.2 images ``` $ guestfish <<_EOF_ > add crc.qcow2 > run > list-filesystems > _EOF_ /dev/sda1: unknown /dev/sda2: ext4 /dev/sda3: xfs ```
This came from https://github.com/coreos/coreos-assembler/pull/737/commits/bbc61e3bc0b1666ea951284010602e5c2308b7e2 > Another thing we were discussing is temporarily removing the immutable attribute Makes sense to me! I don't think there's any RHCOS bug here, CRC should add those libguestfs commands above to their build scripts, right?
(In reply to Colin Walters from comment #9) > CRC should add those libguestfs commands above to their build scripts, right? Mostly done in https://github.com/code-ready/snc/pull/130
@praveen Based on the latest comments and PR to `code-ready`, this doesn't appear to be an actual RHCOS bug anymore. Would you object to closing it?
Yes this can be closed.